Information Architecture:  Big Data "Data Lake" and Data Warehouse - a case for melding the two with Enterprise Information Integration and Semantic Technologies

Data Lake

"In broad terms, data lakes are marketed as enterprise wide data management platforms for analyzing disparate sources of data in its native format," said Nick Heudecker, research director at Gartner recently in an article published on July 28, 2014 (http://www.gartner.com/newsroom/id/2809117).

The concept of data lake, as the article goes on to describe, is to store varied and disparate data (both structured and unstructured) all in one central location.  Nick describes the Big Data in a great metaphor in this article.  So what does this “data lake” do?  The so called “solution” of putting all the disparate data in its native structure and constructs in to a big reservoir or repository allows research and analysis of the data in the data lake by sophisticated professionals with the right tools who know how to manipulate and analyze data. 

So it’s smooth swimming for the data scientists if I can extend the metaphor a bit.  To the rest of us, however, it’s a big ball of muddy swamp that is difficult to wade through.

Lake Superior

So what lacks in the big data lake?

While it is true that data lakes help with the need for increased speed with which data is available in a centralized location, the reality is that they just provide more real time volume of data.  The quality of information that can be gleaned from them can elude us if the insights are hard to find.

Consistency of data coming in to the data lake is one reason why there is questionable quality.  Even the data for a common domain coming from disparate sources can have disparate data structures and architecture  and the underlying model has a tendency to be in a constant flux for a fast moving system.  The reality in large and small enterprises is that the results of how good the insights are will vary quite often.

Data based on such loosely governed models from data sources that use different master data in their native formats will regularly require understanding of the source, the master data of the source and the corresponding data model not only for any given time but for a period of time.  Only then and with the right amount of 'tuning' of the data will the good results ensue.

Another issue is the ‘big ball of mud’ that gets accumulated in the data lake.  Brian Foote and Joseph Yoder way back in 1997 came up with the term “Big Ball of Mud” systems.  They define the term thus: “A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-balling-wire, spaghetti-code jungle”.  Nothing else but a “big ball of mud” describes big data lake more aptly for most of us even though some Big Data practitioners may disagree. For the rest of us, depending on the level of disparity and versatility, the data retrieved from such a data lake is more noise and less information.

Business Data Warehouse

A Data Warehouse is at the other end of the spectrum from Data Lakes when it comes to organining structured data. One is very informal supporting structured and unstructured data, the other mostly formal with well-defined dimensions and marts for structured data. Data Warehouses have been around for several decades now ever since there was recognition in the enterprise that individual data silos were not serving the business well and there was a need to centralize.

The premise of a Data Warehouse is that data mining techniques can be used on current and historical data to discover previously unknown relationships in the data as well as provide centralized reporting and analytics.  Business Data Warehouse (BDW) term, first coined by IBM, originally defined such a usage of the data from this central system as 'business analysis'.  Later on, the term morphed to 'Business Intelligence' or  'Analytics'.  The premise of Data Warehouse is: what good is it to design product features that markets did not care about and what good is a marketing campaign if it did not promote the most profitable products? Business Intelligence capabilities were quickly built in BDWs to provide a better understanding of products, pricing, profitability, operations research, quality, capacity and customer satisfaction as well as for initiatives in logistics and manufacturing.   Data Warehouses certainly solve quite a few essential problems of quality and consistency, help in various functions such as resource planning and analytics by providing the single view of the "truth" - truth as envisioned by the Data Warehouse designers at a certain point in time and validated and accepted by the consumers by the results it produces.  Data Warehouses over the course of their over two-decade long history have been able to deliver on the promise of mining and centralization of disparate data.

Any practitioner, however, who has engaged in building them will tell you that they are costly to stand up and costlier to maintain.  How difficult can it be to come up with building a BDW for two data sources based on an understanding and alignment of two data models, two sets of master data, two data governance models and two underlying design and functional constructs together, one may ask?  Creating a single version of truth from these two data sources, albeit not very difficult, is certainly not trivial.   Now let’s say you add a third data source in the mix - you are now doubling the complexity of the solution as compared to the first two sources.  Add one more and the complexity is quadrupled compared to the first two data sources.  Any additional source adds to this exponential complexity puzzle. The complexity is also an ongoing issue with Data Warehouses – every time there are updates to a data source’s model, its functional usage, its master data or its governance, the analysis of the alignment of that data source with the rest of the data sources in the BDW needs to be revisited.

Additionally as one builds the BDW one must have the data undergo a certain amount of standardization and change from the original source.  The data can forever change because of the metamorphosis that is needed to align with the BDW requirements leading to the occasional irretrievable changes to the source. This change is well understood and manageable initially for fewer sources and for small periods of time but in the long run with large disparate sources this understanding can be lost.  Additional insights when needed in the long run from this permanently changed data can prove elusive.  To top it all, the data has to be highly structured.

Melding the two spectrums with Enterprise Information Integration and Semantic Technology

Big Data is now being discussed in most of the enterprises although survey results still suggest that the production level implementation is less than 10%.  The insights that can come out of Big Data that help the business in various departments are critical to the success of enterprises and are the reason Big Data is implemented.

Watch out for an explosion in the software space in this arena that support Big Data constructs along side traditional Enterprise Information Integration processes.  The level of dynamic integration that will be offered to tune traditional structured data in real-time and process Big Data to extract business insights will be the 'call to duty' of the data insight tools of the near future.

Semantic technologies are providing the extra lift in the data architecture now. Internet standards to describe the data and constructs consistently are providing the necessary common-speak for the data and are being promoted by W3C - the World Wide Web Consortium - the web standards body.  W3C is working with different industries from Health Care to eGovernment, and Energy — to improve collaboration and innovation adoption through Semantic Web technology. Linked data architecture is being built to support the consistent consumption and understanding of the collective data in the various industries.  This common semantic construct can help build and understand competitive insights.

Technologies that can determine smart matches to business problems and reveal insights from the 'big' business data with juxtaposition to the industry insights available from Semantic web are on the verge of flooding the data software landscape and the tools in their arsenal are multi-pronged - melding BDW with Big Data with EII with Semantic web.  Already, ETL tool manufacturers have embedded the technologies to support Big Data/Big Insights using HDFS and MapReduce.  Now these new technologies are laying their sights on real-time Enterprise Information Integration using uniform information representation using Semantic Web.  

The right strategy of extracting the information out of the Big Data will produce the big insights and will require some of these new technologies.  These insights are the foundation blocks of the values that will be extracted by enterprises with a vision to the future with their data initiatives.  Savvy companies will have the right mix to support marketing and business with these new technologies and will need to be successful at extracting values out of their “data lakes” lest those lakes turn into "data swamps".