Date Lake 2.0: Ushering in Digital Transformation

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Digital transformation is all around us. This goes for both digitally born companies, such as the Googles and Amazons of the world, as well as older companies facing new competition, regulations and market disruption. One of the most important aspects of that transformation is changing from a reliance on physical assets to embracing digital information assets and using these to drive your business.

So how can companies successfully achieve digital metamorphosis?

Data is disrupting business models and bringing performance advances. The ability to turn data into assets that the rest of the organization can use in each division, role and function is crucial to achieving success in the changing digital tides. In fact, McKinsey reported that companies with advanced digital capabilities across assets, operations and workforces grow revenue and market shares faster, improve profit margins three times more rapidly than average, and have been the fastest innovators and the disruptors in their sectors.

Achieving this level of acceleration requires a next-generation data lake built for digital transformation so that data no longer goes underutilized.

It’s Time to Get Out of the Murky Waters of Date Lake 1.0

The data lake is the center of the data assets universe. It is the single biggest repository where data converges and assets are created. Yet the data lake of the past few years, or Data Lake 1.0, is not fit to breed digital transformation.

Turning data into an asset requires an efficient production line, but the one most companies employ is inefficient. Companies need a new and improved data production line that brings all the right parties together and enables them to create re-useable, useful and valuable information assets.

Data Lake 1.0 was about filling the data lake and containing data in one place to turn it into an asset eventually. However, shortcomings made that last and most valuable step difficult, if not impossible. Currently, companies waste too much time and resources on creating assets that business analysts need to derive business value.

In the era of Data Lake 2.0, companies can finally become efficient in turning data into assets.

Welcome Data Lake 2.0: Built for Digital Transformation

Our industry likes to use the term “self-service.” But if we look in the mirror, very little of the analytic process is truly self-service. Business analysts today still rely on IT to deliver data. And while the process has gotten better, there still inefficiencies that slow it down.

I once heard an industry analyst use the term “self-sufficient” when it comes to analytics, and I believe that term is a much better descriptor than self-service. IT owns the data and is here to make the business analysts self-sufficient in getting the right type of data. Data Lake 2.0 realizes that goal.

It’s important to note that making business analysts self-sufficient requires cooperation between IT and the business. On one side, IT is pouring data into the lake trying to turn it into something useful. This data integration, preparation and enrichment work is critical. But in the end, the business analysts must be drawn into the process because only they understand what data is most valuable and useful to them to answer the specific business question at hand.

With Data Lake 2.0, a cooperative curation process is foundational for turning data into assets. Data engineers integrate, prepare and organize data. Then business analysts can take the data and explore it in the final crucial step — data refinement. By creating an integrated process directly on the data lake, you

Eliminate time wasted hopping between tools and personas to get the data in the final form the analyst wants.
Make resource utilization more efficient by eliminating data movement between tools.
Allow analysts to explore large, deep and wide datasets without restriction to find the right data to answer the question.
Give business analysts an easy-to-use visual metaphor that is familiar and responsive in exploring the data in any direction.

Just as important, once analysts have explored and refined the data to their needs, they can save it, share results with others and drop it into their favorite tools like Tableau or PowerBI to present it to the business visually.

Free-form data exploration at this massive scale delivers the agility needed to make Data Lake 2.0 the primary production process for information assets. Analysts can explore any path in the data with the ability to fail fast, and explore in other directions until they find exactly what they need. More importantly, they can conduct exploration at the “speed of thought,” producing a data pipeline that creates information assets incredibly fast.

From a tech standpoint, this is very interesting because of the dynamic indexing aspect, which creates indexes on the fly based on the dimensions, metrics and paths analysts want to look at. Previously, dimensional hierarchies and metrics were required and needed to be pre-defined. Paths such as year, day, month and hour, were pre-determined. Now, businesses aren’t limited by hierarchical structure. They can drill across and manipulate different metrics and dimensions (free-form) allowing for unrestrained data exploration.

Data Lake 2.0: The Information Asset Production Line of the Digital Economy

The industrial age has taught us that cooperative, streamlined processes produce assets faster and more effectively. Raw materials are pulled from the available resources and are shaped by multiple people on the line, each with specialized skills, to produce a quality output.

Data Lake 2.0 is the information asset production line of the digital economy era and fuels digital transformation. It is about helping companies convert all the data they have in their lake — their raw materials — into real information assets the business can use.

Effective information asset production requires a joint venture between IT (controls data) and business (understands value). It needs to facilitate a collaborative process directly on the data lake so these two entities can work together to answer key questions. Data Lake 2.0 streamlines the information asset production process so businesses can achieve successful digital transformation.

By John Morrell, ‎senior director, product marketing, Datameer

Photo courtesy of Shutterstock.