Real Time BI is Right Time BI

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Maybe five or six years ago I first heard business intelligence vendors try to explain that not everyone needed real-time data and insights. “What they usually really need is right-time data—the right information to the right person at the right time,” they would say.

A subset of data to a subset of users delivered overnight would solve the problem at reasonable cost, they insisted.

No one wanted to deal with the cost and complexity of revamping billions of dollars of online transaction processing (OLTP) systems and converting them to online analytical processing (OLAP). The vendors insisted that the traditional approach – bolting a data warehouse onto the OLTP environment to provide users with reports and an occasional OLAP engine for real time info – was an effective and economically efficient solution.

Forgetaboutit, as they say in New York. A combination of new technologies, user expectations based on the consumerization of IT, and accelerated business conditions has made real time the new right time for a large swath of business users, as well as for IT itself.

A new series of surveys – plus numerous conversations with IT industry officials – makes it clear that IT professionals must rethink their traditional response when a manager demands real time access to BI or analytics systems.

Forty percent of respondents to a recent survey I managed said their organizations would have real-time access to analytics and business intelligence information instead of batch reports within two years. (Survey included more than 300 corporate executives).

Furthermore, real-time reporting was the number one investment priority among 10 different technology projects including analytics, consolidating data centers, Big Data and mobile access (see table for a sample of their answers):

real time business intelligence

Note that another 38% were already getting reports in real time rather than batch and 39% were already providing mobile device access to financial data and systems.

Recent surveys by Ventana Research, a pre-eminent BI/analytics benchmark research and advisory services firm, found even stronger interest in real time BI and analytics. Almost six out of ten respondents (mostly business users) to its Operational Intelligence Benchmark Research project said managing performance, detecting fraud, or compliance demands drove their demand for real time (see table):

real time business intelligence

Source: Ventana Research Operational Intelligence Benchmark Research

Ventana’s recent Business Technology Innovations Benchmark Research, mostly of IT officials, also indicated strong interest in real time BI/analytics for security, network and systems management (see table):

real time business intelligence

Source: Ventana Research Business Technology Innovations Benchmark Research

Tony Cosentino, VP & Research Director of Ventana Research, noted that the survey found even stronger interest in implementing real time BI/analytics in the next two years via complex event processing. “We are currently past the early adopter market for event processing, with 18% already having the capability,” he said. “Another 29% of survey respondents said they would have it within the next 12 months and an additional 49% said they would have it within the next 24 months.”

Furthermore, he added, “2013 will see a much broader discussion about real time BI and analytics.”

Indeed, much of the discussion will be to clarify terms. Real time BI/analytics means different things to different people. The needs vary by department, industry, and national and corporate culture.

“There is still a lot of confusion over what right time and real time really are,” noted Cosentino. “When we talk about real-time transaction processing, with data streaming into mobile devices right from transaction systems, rules-based analytics trigger actions. This complex event process is like when a car crashes and the air bags deploy. These rule-based systems are different than the complex algorithms that may be at work behind the scenes.”

In addition to the immediate reaction to an event, Cosentino and others warn IT that everybody expects everything right away. “If someone waits more than two seconds for something to appear on their screen, they will jump to another screen,” Cosentino added.

For more information about the Ventana Research data, here’s a video of a Cosentino presentation to clients.

The impatience to get data and analytics fast is one of the byproducts of the consumerization of IT. Michael Driscoll, CEO of cloud analytics provider Metamarkets, summarizes customer demands this way: “We’re entering the Age of Enlightenment for enterprise IT. Executives are starting to ask ‘Why does it take us days to get an internal report, when Google tells me instantly what is out there? Why are our systems slow but others are fast?'”

IT professionals who balk at delivering real time BI/analytics, citing standard explanations for the delays and standard caveats about moving to a real time environment, are doomed. Senior business execs will reject warnings that the existing IT infrastructure was built for batch mode and that the costs and complexities to shift to real time are prohibitive.

Francois Ajenstat, Director of product management at Tableau and a veteran of 15 years in the BI space at Cognos and Microsoft, advocates a different approach. IT pros should take advantage of the faster database software, faster processors, cheaper memory and other enablers and acknowledge that the choice is not either/or when it comes to real time vs. batch.

“Organizations have a need for batch and for real time – they have a need for all of these capabilities,” Ajenstat said. “The challenge becomes how to ensure that users get what they want when they want it.”

I’m going to go even further. Here are three guiding principles for IT professionals to embrace that will ensure their customers’ satisfaction and prolong their careers:

• Assume new implementations will be real time.

• Immediately chart a course to add real-time capabilities to your top tier user and customer-facing applications within the next 12 months.

• Review the existing IT architecture and standards and pull back from the rigid controls on the number of “approved” databases, BI and analytics tools.

I know you think I’m advocating letting the inmates run the prison and that I’m just promoting what the vendors would like to see happen. The reality is that your end users are adopting cloud-based analytics and other low cost/low threshold-of-entry tools without your knowledge, let alone approval. The recent announcement from Amazon that its Web services unit now includes analytics (the Redshift service) is just the tip of the iceberg of what is out there or will be soon.

Tableau’s Ajenstat describes a scenario that, while benefiting his company and other vendors, reflects the pragmatism that IT professionals should adopt given the proliferation of needs and alternative solutions: “Empower the business side with whatever they need. IT should say, ‘we will focus on infrastructure, not the entire end to end system. You can use whatever tools on top of the data.’ IT used to try to put everything in the data warehouse, but that is too confining and slow given the current environment. Now IT should adopt a multi-tool strategy.”

Your users already have, you just don’t know it.