The In-Memory Database Revolution

Carl Olofson

In-Memory Database RevolutionThis is a seminal moment in the history of database technology: a moment when the dominant paradigms for database management for the past 40 years are being challenged by new approaches designed to take advantage of changes in system power and architecture and shifts in the underlying cost structure.

We are seeing the convergence of very fast, multi-core processors, lower cost main memory, and fast, configurable networks with demand for extreme transaction rates, high speed complex queries, and operational flexibility. One area of software technology that has arisen in response to this convergence is memory-base database management.

Unlike disk-based database management, memory-based technology does not require optimization for disk storage, has no overhead for such optimization, and dramatically reduces the storage footprint while simultaneously delivering extremely high throughput rates.

Memory-based databases are optimized for manipulation in memory, with less frequently accessed data swapped out to disk. Not all memory based databases require disk swapping, however. The fastest form of this technology is that which holds the entire database in memory all the time; this is commonly called in-memory database (IMDB) technology.

The implications of this new technology are broad and varied. We are just seeing a glimpse of what may be done when all the data is managed in memory. Because reorganizing data on disk is slow and cumbersome, and because supporting alternate forms of access adds unacceptable overhead, disk-based databases have required fixed schematic structures that afford only a single mode of access, usually involving base tables, with views defined to offer a bit of access flexibility.

IMDB, on the other hand, allows data to be dynamically reorganized, and viewed according to multiple paradigms. As a result, an IMDB can handle on-the-fly schema changes, and in many cases can render the same data either in conventional relational table form, or as complex objects or documents as required.

Operationally, the implications are just as profound. Database administrators no longer need to spend most of their time pondering storage allocation, index definition, and scheduling unload/reloads for data reorganization and re-indexing.

They can, instead, concentrate on building data structures and renderings that address the business needs of the enterprise, and provide higher value support to applications and users; the kind of support that gains recognition and yields professional rewards.

A number of new database technology firms have emerged over the past few years, delivering IMDB products optimized for various workloads. Some more established firms have joined in, offering new IMDB technologies that promise to disrupt the database technology marketplace. One that is already making waves with its ability to mix transactional and analytic workloads is SAP HANA, part of SAP’s “Real Time Data Platform”.

The “old guard” vendors also are evolving their technology feverishly in an IMDB direction. Anyone involved in database technology from either a data or application management perspective would be well advised to learn about these companies and initiatives; they are the future of this business. Embrace the new paradigm, and plan for it!

For more information, listen to a replay of the webinar entitled “The Key to Running in Real-Time:  In-Memory Database Technology”.