Big Data just keeps getting bigger, in a popularity sense. A new IDC report predicts that the Big Data and business analytics market will grow to $203 billion by 2020, double the $112 billion in 2015.
The banking industry is projected to lead the drive and spend the most, which is not surprising, while IT and businesses services will lead most of the tech investing. Overall, IDC finds that the banking, discrete manufacturing, process manufacturing, federal/central government, and professional services will account for about 50% of the overall spending.
Not surprisingly, some of the biggest big data analytics spending — about $60 billion — will go toward reporting and analysis tools. That’s what analytics is all about, after all. Hardware investment will reach nearly $30 billion by 2020.
So as Big Data grows, what will be the major trends? In talking to experts and surveying the research reports, a few patterns emerged.
1) Simplified analytics: While the work of Big Data will grow increasingly complex, the software behind it will advance to better handle multiple varied and complex data sets, without needing a data science degree.
2) Machine Learning: Big Data solutions will increasingly rely on automated analysis using machine learning techniques like pattern identification and anomaly detection to sort through the vast quantities of data.
3) Predictive analytics: Machine learning is not just for historical analysis, but also can be used to predict future data points. That will start with basic ‘straight-line’ prediction, deducing B from A. But it will eventually grow and become more sophisticated by detecting patterns and anomalies that are about to happen too.
4) Security analytics: To some degree this aready has a significant prescence. Security software, especially intrusion detection, has learned to spot suspicious and anomalous behavior. Big Data, with all of its source inputs, needs to be secured and there will be greater emphasis on securing the data itself. The same processing power and software analytics used to analyze the data will also be used for rapid detection and adaptive responses.
5) The bar is raised: Traditional programmers will have to add gain data science skills to their repertory in order to stay relevant and employable. But just like many programmers are self-taught, there will be a rise in data scientists from nontraditional professional backgrounds, including self-taught data scientists.
6) The old guard fades: A 2015 report from Gartner found Hadoop was fading in popularity in favor of real-time analytics like Apache Spark. Hadoop was, after all, a batch process run overnight. People want answers in real time. So Hadoop, MapReduce, HBase and HDFS are all going to continue to fade in favor of faster technologies.
7) No more hype: Big Data has faded as a buzzword and is now just another technology like RDBMS and CRM. That means the technology has settled into the enterprise as another tool brought to bear. It’s now a maturing product, free of the hype that can be distracting.
8) More Data Scientists: The Data Scientist is probably the most in-demand technologist out there, with people who qualify commanding a significant salary. Nature abhors a vacuum and you will see more people trying to gain Data Scientist skills. Some will go the self-taught route, which is how many programmers acquired their skills in the first place, while others will get training via crowdsourcing.
9) IoT + BD = soulmates: millions of Internet-connected devices, from wearables to factory equipment, will generate massive amounts of data. This will lead to all kinds of feedback, like machine performance, which in turn will lead to optimized performance and earlier warnings before failure, reducing downtime and expenses.
10) The lake gains power: Data lakes, massive repositories of information, have been around for a while but mostly it’s just a store with little idea how to use it. But as organizations demand quicker answers, they will turn to the data lake for those answers.
11) Real time is hot: In a survey of data architects, IT managers, and BI analysts, nearly 70% of the respondents favored Spark over MapReduce. The reason is clear: Spark is in-memory, real time stream processing while MapReduce is batch processing usually done overnight or during off-peak hours. Real-time is in, hours-old data is out.
12) Metadata catalogs: You can gather a lot of data with Hadoop but you can’t always process it, or even find what you need in all that information. Enter Metadata Catalogs, a simple concept where aspects of Big Data analytics, like data quality and security, are stored in a catalog. They catalog files using tags, uncover relationships between data assets, and even provide query suggestions. There are a number of companies offering data cataloging software for Hadoop, plus there is an open source project, Apache Atlas.
13) AI explodes: Artificial intelligence, and its cousin machine learning, will see tremendous growth because there is simply too much data coming in to be analyzed to wait for human eyes. More must be automated for faster responses. This is especially true with the massive amounts of data generated by IoT devices.
14) Dashboard maturity: With Big Data still in its early years, there are a lot of technologies that have yet to mature. You just can’t rush some things. One of them is the right tools to easily translate the data into something useful. Analysts predict that dashboards will finally get some attention from startups like DataHero, Domo, and Looker, among others, that will offer more powerful tools of analysis.
15) Privacy clash: With all the data being gathered, some governments may put the brakes on things for a variety of reasons. There have been numerous government agency hacks and questions about the 2016 Presidential election. This may result on restrictions from the government on how data is gathered and used. Plus, the EU has set some tough new privacy laws regarding how data is used and how models are built, set to take effect in January 2018. The impact is not yet known, but in the future, data might be harder to come by or use.
16) Digital assistants: Digital voice assistants like Amazon Echo and Alexa and Google Home and Chromecast will be the next generation of data gathering, along with Apple Siri and Microsoft Cortana. Don’t think they won’t. These are “always listening” devices used to help people make purchase and other consumption decisions. They will become a data source at least for their providers.
17) In-memory everything: Memory has up to now been relatively cheap, and since 64-bit processors can access up to 16 exabytes of memory, server vendors are cramming as much DRAM into these things as possible. Whether in the cloud or on-premises, memory footprints are exploding, and that’s making way for more real-time analytics like Spark. Working in memory is at three orders of magnitude faster than going to disk and everyone wants more speed.
Huawei’s AI Update: Things Are Moving Faster Than We Think
FEATURE | By Rob Enderle,
December 04, 2020
Keeping Machine Learning Algorithms Honest in the ‘Ethics-First’ Era
ARTIFICIAL INTELLIGENCE | By Guest Author,
November 18, 2020
Key Trends in Chatbots and RPA
FEATURE | By Guest Author,
November 10, 2020
FEATURE | By Samuel Greengard,
November 05, 2020
ARTIFICIAL INTELLIGENCE | By Guest Author,
November 02, 2020
How Intel’s Work With Autonomous Cars Could Redefine General Purpose AI
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
October 29, 2020
Dell Technologies World: Weaving Together Human And Machine Interaction For AI And Robotics
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
October 23, 2020
The Super Moderator, or How IBM Project Debater Could Save Social Media
FEATURE | By Rob Enderle,
October 16, 2020
FEATURE | By Cynthia Harvey,
October 07, 2020
ARTIFICIAL INTELLIGENCE | By Guest Author,
October 05, 2020
CIOs Discuss the Promise of AI and Data Science
FEATURE | By Guest Author,
September 25, 2020
Microsoft Is Building An AI Product That Could Predict The Future
FEATURE | By Rob Enderle,
September 25, 2020
Top 10 Machine Learning Companies 2020
FEATURE | By Cynthia Harvey,
September 22, 2020
NVIDIA and ARM: Massively Changing The AI Landscape
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
September 18, 2020
Continuous Intelligence: Expert Discussion [Video and Podcast]
ARTIFICIAL INTELLIGENCE | By James Maguire,
September 14, 2020
Artificial Intelligence: Governance and Ethics [Video]
ARTIFICIAL INTELLIGENCE | By James Maguire,
September 13, 2020
IBM Watson At The US Open: Showcasing The Power Of A Mature Enterprise-Class AI
FEATURE | By Rob Enderle,
September 11, 2020
Artificial Intelligence: Perception vs. Reality
FEATURE | By James Maguire,
September 09, 2020
Anticipating The Coming Wave Of AI Enhanced PCs
FEATURE | By Rob Enderle,
September 05, 2020
The Critical Nature Of IBM’s NLP (Natural Language Processing) Effort
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
August 14, 2020
Datamation is the leading industry resource for B2B data professionals and technology buyers. Datamation's focus is on providing insight into the latest trends and innovation in AI, data security, big data, and more, along with in-depth product recommendations and comparisons. More than 1.7M users gain insight and guidance from Datamation every year.
Advertise with TechnologyAdvice on Datamation and our other data and technology-focused platforms.
Advertise with Us
Property of TechnologyAdvice.
© 2025 TechnologyAdvice. All Rights Reserved
Advertiser Disclosure: Some of the products that appear on this
site are from companies from which TechnologyAdvice receives
compensation. This compensation may impact how and where products
appear on this site including, for example, the order in which
they appear. TechnologyAdvice does not include all companies
or all types of products available in the marketplace.