By Dmitri Tverdomed
Traditional data warehousing has several drawbacks than can prevent businesses from getting true value out of ever growing data sets - are data lakes the solution?
The risks oftraditional data warehousing are spreading. As more companies source their own business data solutions, data warehouses became the norm - single repositories for all of the information pertaining to the operations and consumer base of a company with a rigid, highly governed, conformed data model.
But with exponential growth in data volumes (and the actionable insights that can be derived from it), traditional data warehousing may not be the best solution for many businesses. This is where a more flexible data lake or data hub can prove useful.
Keeping up with rapid expansion
Many large traditional data warehousing solutions use a unified reference data model, which is too rigid and time-consuming to adapt. Available data is increasing at an exponential rate, meaning every week companies have new variables or tweaks that needs to be made to their business analytics in order to get true value.
However, traditional warehousing was not designed for this rapid turnaround of information. While warehouses will prove functional for a huge number of companies, many still need a more fluid, adaptable system for efficient analytics.
Flexible data architecture for growing businesses
Disruptors are everywhere. Amazon has turned the retail market upside down, while the financial services sector has pushed back strongly against the proliferation of mobile systems like Apple Pay. Nonetheless, tech keeps advancing and disruptors keep emerging.
Traditional data warehousing simply can't keep up on this front. Every time a market is disrupted, there is an entirely new kind of information that needs to be collected and analysed. Older warehouse models may not be able to keep up with this intake, leaving businesses needing something more.
Data lakes: A solution, but not the solution
The data lake model can be used to avoid the restrictive ‘conformity’ of traditional warehousing.
Data lakes operate with flat architecture, meaning information is stored in a ‘true-to-source’ structureless environment. This opens up the breadth of information businesses can ingest efficiently, giving companies greater flexibility in their business analytics.
The flexibility of data lakes can enhance a business' analytical capacity.
Through this model, businesses can avoid bottlenecks that occur when new datasets arise that don't fit into the enterprise confirmed data model. However, it also means information is stored but not managed - businesses require a comprehensive analytic platform to make sense of the lake.
Furthermore, this flat architecture necessitates some more forethought with the way data is managed through its lifecycle. Metadata, lineage, business context and data asset cataloguing become critical - without these, data lakes can become impossible to navigate, essentially becoming data swamps.
Every company has different needs - while traditional data warehousing will still work (and continue to for a long time), companies with more diverse datasets and rapid expansion in the pipeline may need the flexibility that a lake offers. No matter what kind of data architecture you require, you're going to need the right tools to process it - which is where the business intelligence experts at Cornerstone can help.
If you'd like to discuss this further please contact Steven Gill at firstname.lastname@example.org or on 0421 566 219.