Building The Big Data Warehouse: Part 1

Barbara Lewis

Part 1 in the “Big Data Warehouse” series

The enterprise data warehouse (EDW) architecture has long been a key technology asset for fast analytics on cleansed, curated, and structured business data. It is a critical technology foundation of many enterprises. However, it is straining to deliver value in the era of exploding data volumes and increased demand for analytics and data across the organization.

Not only are the sources of structured, traditional data increasing in volume – such as transactional, operational, and financial information – but organizations are also embracing the age of Big Data, dealing with new types of data that the enterprise data warehouse was not built to handle. This includes unstructured information, like weblog and machine log information, audio, video, and social media interactions; high-speed information, like sensor data in Internet-of-Things scenarios; and third-party information, like weather, public databases, or brokered information. All of this data is being introduced to the enterprise at unprecedented volume and speed. This Big Data is stored in systems uniquely capable of handling them, such as Hadoop-based data lakes or cloud object storage.

The business potential inherent in all of this data is demanding to be tapped, not only to improve the efficiency and quality of existing goods and services, but also to create new offerings or business models that can accelerate an organization ahead of the competition. However, in order to achieve this, enterprises need a way to interconnect Big Data with enterprise data. They also need a way to provide the analytical responsiveness, security, and ease of use that are associated with the enterprise data warehouse and its applications.

Why a Big Data warehouse?

With a Big Data warehouse approach, companies are looking to:

  • Leverage existing investments made in technology, processes, and people. While organizations recognize the limitations of the EDW in the face of new data demands, they also recognize the value of the data already being managed effectively by the EDW and want to leverage this as part of their new enterprise architecture. There is not only the financial investment in the technology itself to consider, but also the value of the existing processes that are well understood by the enterprise’s employees, as well as partners and vendors. While change and new investment is inevitable, leveraging the EDW to its maximum potential makes clear sense from a financial and change management perspective.
  • Innovate for the future, leveraging new and faster sources of data. Enterprises are increasingly embracing Big Data, especially as Big Data solutions become easier to use, more secure, and better integrated to traditional enterprise systems. These new, faster data sources represent greater opportunities for improving existing products and services, as well as an opportunity to capture new and emerging markets. Enterprises across industries increasingly see competitive threats from disruptive new entrants that are effectively using new data architectures built for Big Data from the start.
  • Make faster, more responsive, and even proactive decisions. End customer expectations for service speed, corporate responsiveness, and information-sharing from the companies that serve them are increasing quickly. As a result, enterprises are looking to deliver on expectations by accelerating their own speed of data collection, processing, and analysis, aiming to spread “right time” data and decision making as broadly across their organizations as they can. Forward-looking organizations have future goals not only for responsiveness to rapidly changing market or customer conditions, but also for achieving predictive analytics that can help them to address issues before they escalate, or spot trending opportunities in their earliest stages.
  • Empower more managers and decision-makers with analytics. “Self-service” analytics has been a goal of enterprises for some time now, and the intensity of pressure for achieving this goal has only increased. Organizations are realizing that limiting analytics and decision making to an elite few requires too much time and can squander opportunities or exacerbate emerging challenges. By more broadly distributing information and analysis to a wider range of managers and decision-makers, including partners and key vendors, the organization becomes more responsive and agile.
  • Ensure enterprise-class security and data governance, even as data volumes grow and analytical end users proliferate. While Big Data solutions have long offered a scalability and data diversity advantage that EDWs could not match, there have also been longstanding concerns about data security and data governance for these emerging technologies. As the technologies mature and can increasingly meet the security needs of the most demanding organizations, there is increased willingness to let Big Data projects out of the “lab” and interconnect them with the broader enterprise data architecture and its larger group of end users.

With this perspective on the reasons behind the new Big Data warehouse architecture push and what enterprises hope to gain from it, Part 2 of this four-part series will cover the key elements of a Big Data warehouse and the key issues to keep in mind when selecting critical technology components for the new architecture.

Learn more

Barbara Lewis

About Barbara Lewis

Barbara Lewis is the VP of Marketing for SAP Cloud Platform Big Data Services and a thought leader in SAP’s Big Data practice, with expertise in cloud, Big Data solutions, data landscape management, Internet of Things (IoT), analytics, and business intelligence. Barbara led the launch of SAP Data Hub, the latest Big Data offering from SAP, and is active in SAP’s Big Data Warehousing initiative.