Big Data Analytics Is Not a Niche Anymore

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

In case you were wondering if everyone else was stunned by the rapid rise of big data as a normal topic of business and IT conversations, the fast answer is yes. I, too, was surprised how last year’s buzzword du jour became Standard Operating Procedure for so many companies in so short a time.

A survey a few months ago by The Data Warehousing Institute (TDWI) and a report by its research director, Philip Russom, paints a picture of a nascent trend going mainstream faster than you could say “Siri combines artificial intelligence and a search engine.”

More than 90% of the 325 IT professionals and business sponsors/users responding to the survey said they are familiar with the concept of big data analytics. Big data analytics is the term to describe subjecting many terabytes of structured and unstructured data to near real-time analysis by advanced software tools using sophisticated algorithms to find patterns and therefore opportunities.

Big data analytics is a lot bigger and more complicated than your father’s data mining. It’s been more than a decade since data mining, data clustering, market basket analyses and other related Business Intelligence/analytics tasks developed into narrow niche fields known to a relative handful of practitioners. Contrast that to the one third of the survey respondents who said they are actually engaged in big data analytics (see table):

big data analytics

Source: TDWI

It isn’t quite as dramatic as a perfect storm of good things happening, but a confluence of technologies are enabling organizations to actually use those terabytes of emails, blogs, tweets, videos and other information outside of a database.

“User organizations have been collecting Web data for years,” noted Russom. “For most organizations, it’s been a kind of hoarding. We’ve seen similar untapped big data collected and hoarded, such as RFID data from supply chain applications, text data from call center applications, semistructured data from various business-to-business processes, and geospatial data in logistics.”

What’s changed, Russom explained, is that “far more users are now analyzing big data instead of merely hoarding it.” And thanks to data warehouse appliances and software capable of processing the huge data flows, organizations can now find, polish and use the gems buried in the flow of jetsam swirling around their data centers.

Of course, those gems are typically the insights that many users and all the business intelligence vendors touted for their products. Better business decisions, the ability to identify disaffected customers ready to bolt (known as churn in the telecom industry), and pricing optimization by the airlines and other industries were typical of the benefits that accrued to those organizations spending big bucks for the systems.

Interestingly, the TDWI survey found those types of jewels weren’t at the top of their big data analytics expectations. Instead, it appears that big data analytics in many cases will be applied to a new challenge, social media analysis.

As you can see in the following table, the expected beneficiaries of a big data analytics initiative is a combination of what has been promised in the past as well as dealing with the current tweet, blog and YouTube challenge of today:

When asked, “which of the following benefits would ensue if organization implemented some form of big data analytics (pick no more than 5)?” here are the survey respondents’ answers:

Source: TDWI

All of these benefits are predicated on being able to implement a big data analytics program, and there’s a significant challenge. Specifically, a shortage of qualified big data people.

Currently the hot title/job description is data scientist. Even Fortune magazine’s web site recently posted articles about the huge spread between data scientist supply and demand.

The shortage of data scientists is the number one barrier to organizations jumping on the big data analytics bandwagon, according to survey respondents (see table). Note that the people shortage is an even bigger deal than the cost of the machines and software involved.

What are the top potential barriers to implementing big data analytics?

Source: TDWI

It is important to note that the skills and dollars shortage are not considered deal breakers for most companies. Only 30% of the survey respondents considered big data analytics a problem for the organization—70% considered it an opportunity.

That’s probably the most refreshing and pleasant surprise about the survey—IT is embracing the big data analytics challenge rather than trying to avoid it.