Need Real-Time Insight? Bring Live And Historical Data Together Again

Stefan Baeuerle

It’s hardly news that comprehensive access to all relevant data is a critical first step toward the single source of truth that organizations need to succeed in the digital economy. To increase insight, serve customers better, speed response times, and improve business performance, you need to connect to all data from the far-flung corners of the enterprise and consolidate it into a single view. Otherwise, the business is left making decisions based on only part of the story.

But even for organizations that make comprehensive data access a priority, there’s still one split in the typical data environment that stands as a persistent obstacle to the single source of truth: the split between live data and historical data.

What’s the difference?

Live data is about what’s going on in your business right now. Think of customer interaction data, point-of-sale transactions, IoT sensor streams, or Tweets. Historical data is the same as live data, except that it’s old. But old isn’t bad. Old, historical data, in fact, is quite relevant and necessary for identifying patterns and trends.

The issue is that live and historical data are typically stored in separate systems. It wasn’t always this way, though. Years ago, they lived together in the same system – but as data volumes grew, performance suffered. To optimize performance, organizations split them apart – keeping live data in transaction-processing systems dedicated to speed and historical data in data marts and warehouse systems dedicated to analytics.

The ramifications of a split data environment

One result of this decision was a lot of administrative overhead for IT. To analyze live and historical data together, IT got busy extracting data from one system, loading it into another, and transforming the data for analysis – while ensuring data quality and controlling for redundancies.

Another result was a delay in the speed of insight from all this data. With analytics conducted on a separate system, decision timeliness depended on how fast your IT team could batch-load live data into data warehouses for analysis.

In a real-time digital economy, however, there is little time to wait, and insight from yesterday won’t do. Increasingly, organizations need insight in the moment. To detect patterns and reveal insight, live data needs to be analyzed instantaneously and in context. For this to happen, the two kinds of data need to be brought together again.

A new kind of data platform

But wouldn’t processing new transactions and analyzing data in one system just bring us back to the original problem of poor performance? Not if you run on a hybrid transactional/analytical processing (HTAP) platform with the speed of in-memory processing.

As the name implies, an HTAP data platform brings transactional and analytical processing together on a single system. The added advantage of in-memory processing means great performance. Instead of retrieving data from disk, the data is accessible and available in memory almost instantaneously, allowing you to perform more complex tasks much faster. Now you can capture incoming business transactions, combine that with historical data, and extract insights in real time. Such a platform can also process different types of data in conjunction – combining, say, business data with location and text data to provide deeper insights, predict business outcomes, or suggest a new course of action.

But how do you hold today’s ever-growing volumes of data in memory? An HTAP platform that combines in-memory processing with a columnar data store can help. By storing your data in columns, you can achieve a higher level of data compression so that you can fit more data in memory. You can also get answers to questions faster because the columnar store can retrieve just the data attributes that are relevant for a given question and skip the rest. In another blog, Carl Dubler does a really good job of explaining the advantages of columnar storage compared to other data management approaches.

Sometimes, despite the high degree of compression, the size of your data might exceed the memory capacity of your hardware. This is why storage flexibility is another important consideration. What’s needed is an HTAP platform that maintains all the important data in memory by default, but can also work efficiently with disks for data that is accessed less frequently. Importantly, though, even the data stored on disk remains very much accessible – only with a bit more latency.

Now, when you combine in-memory data management and the right data-storage strategy, you’re processing more data at orders of magnitude faster.

The benefits of speed

What does this speed of processing mean? It means that now all data – live and historical – can be processed together without poor performance standing in the way. If you need to record new business transactions and analyze them together with historical data, you no longer need to do the analysis on a separate system. Now you can just run a query against the single source of truth you’ve created – in virtually no time at all.

The benefits of such an HTAP platform helps your business in numerous ways. One example is the financial close. In the past, monthly, quarterly, or yearly close processes would consume finance and accounting groups for days on end. But now, with the speed of an in-memory HTAP data platform, you can achieve the “continuous close” – where up-to-date financial data is made available on demand.

Performance gains also give your business the power to benefit from artificial intelligence techniques such as machine learning. With powerful machine-learning algorithms that use live and historical data, you can detect patterns, identify anomalies, and accurately predict business outcomes. Most importantly, you can leverage these insights in your business processes to guide your users and improve decision-making. This helps to arm your business with full situational awareness and the ability to sense and respond in the moment. In the end, you’re able to continuously improve business performance, enhance the customer experience, and move closer to the goal of becoming an intelligent enterprise. Not bad!

Learn more

Click here to find out more about intelligent data management.


Stefan Baeuerle

About Stefan Baeuerle

With a history of more than 20 years at SAP in different roles in software development, architecture, and management, Stefan Baeuerle is a development executive expert for the SAP Database and Data Management Unit, as well as a member of the SAP CTO Circle and the CTO Circle Decision Board. In his work, Stefan focuses on various aspects of data management and the SAP HANA platform, including its use across diverse scenarios, related programming models, and infrastructures. He is currently working on extending data management into multi-model scenarios, spanning relational and NoSQL data formats, machine learning, cloud infrastructures, and managed SAP HANA as a Service in the cloud, as well as the overall SAP HANA platform strategy.