Next-Generation, Real-Time Data Warehouse: Bringing Analytics To Data

Iver van de Zand

Imagine the following situation: you are analyzing and gathering insights about product sales performance and wonder why a certain area in your country is doing better than others. You deep dive, slice, dice, and use different perspectives to analyze, but can’t find the answer to why sales are better for that region.

You conclude you need data that is not available in your corporate systems. Some geographical data that is available through Hadoop might answer your question. How can you get this information and quickly analyze it all?

Bring analytics to data

If we don’t want to go the traditional route of specifying, remodeling the data warehouse, and uploading and testing data, we’d need a whole new way of modern data warehousing. What we ultimately need is a kind of semantics that allows us to remodel our data warehouse in real time and on the fly – semantics that allows decision makers to leave the data where it is stored without populating it into the data warehouse. What we really need is a way to bring our analytics to data, instead of the other way around.

So our analytics wish list would be:

  • Access to the data source on the fly
  • Ability to remodel the data warehouse on the fly
  • No replication of data; the data stays where it is
  • Not losing time with data-load jobs
  • Analytical processing done in the moment with pushback to an in-memory computing platform
  • Drastic reduction of data objects to be stored and maintained
  • Elimination of aggregates

Traditional data warehousing is probably the biggest hurdle when it comes to agile business analytics. Though modern analytical tools perfectly add data sources on the fly and blend different data sources, these components are still analytical tools. When additional data must be available for multiple users or is huge in scale and complexity, analytical tools lack the computing power and scalability needed. It simply doesn’t make sense to blend them individually when multiple users require the same complex, additional data.

A data warehouse, in this case, is the answer. However, there is still one hurdle to overcome: A traditional data warehouse requires a substantial effort to adjust to new data needs. So we add to our wish list:

  • Adjust and adapt the modeling
  • Develop load and transformation script
  • Assign sizing
  • Setup scheduling and linage
  • Test and maintain

In 2016, the future of data warehousing began. In-memory technology with smart, native, and real-time access moved information from analytics to the data warehouse, as well as the data warehouse to core in-memory systems. Combined with pushback technology, where analytical calculations are pushed back onto an in-memory computing platform, analytics is brought back to data. End-to-end in-memory processing has become the reality, enabling true agility. And end-to-end processing is ready for the Internet of Things at the petabyte scale.

Are we happy with this? Sure, we are! Does it come as a surprise? Of course, not! Digital transformation just enabled it!

Native, real-time access for analytics

What do next-generation data warehouses bring to analytics? Well, they allow for native access from top-end analytics components through the data warehouse and all the way to the core in-memory platform with our operational data. Even more, this native access is real-time. Every analytics-driven interaction from an end-user generates calculations. With the described architecture, these calculations are massively pushed back to the core platform where our data resides.

The same integrated architecture is also a game changer when it comes to agility and data optimization. When new, complex data is required, it can be added without data replication. Since there is no data replication, the data warehouse modeling can be done on the fly, leveraging the semantics. We no longer have to model, create, and populate new tables and aggregates when additional data is required in the data warehouse, because there are no new tables needed! We only create additional semantics, and this can be done on the fly.

Learn why you need to put analytics into your business processes in the free eBook How to Use Algorithms to Dominate Your Industry.

This article appeared on Iver van de Zand.


Iver van de Zand

About Iver van de Zand

Iver is the Director of  the SAP Global Analytics Hub for business intelligence and predictive analytics focusing on enablement for pre-sales, collaboration, content generation, and best practices. He works closely with global leadership and stakeholders across SAP incorporating the latest insights, tools, and best practices in order to optimize the use of SAP resources, improve cross organisational collaboration, and drive efficiencies in business execution. Iver is also a member of the Lumira Advisory Council (LAC) and the International Business Communication Standards (IBCS) community that focuses on data visualization standards and Hichert principles.