Disposable Technology: A Concept Whose Time Has Come

Bill Schmarzo

Imagine that you have been challenged to play Steph Curry, the greatest three-point shooter in the history of the National Basketball Association, in a game of one-on-one. Yeah, a pretty predictable outcome for 99.9999999% of us.

But now imagine that Steph Curry has to wear a knight’s suit of armor as part of that game. The added weight, the obstructed vision, and the lack of flexibility, agility, and mobility would probably allow even the average basketball player to beat him.

Welcome to today’s technology architecture challenge!

For many companies, their technology architecture is becoming more of a hindrance than an enabler of value creation. It resembles an archeological dig, with layers of ancient technologies layered on top of even more ancient technologies. The result: added weight, obstructed vision, and lack of flexibility, agility, and mobility.

Modern digital companies like Google, Facebook, Twitter, Apple, Netflix, Amazon, and Airbnb have taken a technology architecture approach that increasingly treats the technology infrastructure as “disposable,” using open source technologies.

And the reason for this open approach, in my humble opinion, is twofold:

  • First, building upon open source technologies provides the flexibility, agility, and mobility for companies to move to the next-best technology without the constraints and architectural lock-in of traditional technology. Traditional technologies advance at the rate of the proprietary technology vendors. These vendors dictate new technology capabilities on a rigid, unbending nine- to 12-month product release schedule for which customers get the honor of paying a 30%+ annual maintenance fee. Modern digital companies are basing their technology infrastructure on open source technologies that not only prevent vendor architectural lock-in, but also allow them to advance the technology capabilities at their pace and at the pace of the business.
  • Second and more important, these digital companies understand that the technology isn’t the source of business value and differentiation. They understand that the source is:
    • The data that these organizations are masterfully amassing via every customer engagement and every usage of the product or service
    • The customer, product, and operational insights (intellectual property in the form of customer, product, and operational propensities, tendencies, associations, relationships, and patterns) that lead to new monetization and commercialization opportunities

Let’s drill into these two tenets to see the takeaways that can be replicated by any company interested in digitally transforming its business models.

Understanding the modern digital business architecture

Digital companies understand that trying to compete using traditional technologies is like Steph Curry being forced to wear knight’s armor while competing in a basketball game. For a great lesson in modern technology, let’s look at Lyft’s strategy for structuring its technology architectures (from the article “What’s Behind Lyft’s Choices in Big Data Tech”).

The company:

  1. Started with AWS Redshift, but transitioned to Apache Hive when it started to run into scalability issues
  1. Migrated to Presto to provide a more powerful query engine that supports data exploration analytics across multiple data sources
  1. Uses Apache Spark to support massive ETL batch processing and to train its machine learning models
  1. Uses Druid, a column-oriented, in-memory OLAP data store that excels at performing drill-downs and roll-ups over a large set of high-dimensional data
  1. Uses Jupyter (data science team), a popular notebook-style interface for working with data and machine learning algorithms, and the PySpark library
  1. Uses Apache Airflow, which creates repeatable data engineering and data science workflows that can be executed atop the workflow orchestration tool Kubernetes
  1. Uses a mixture of Apache Kafka, Apache Flink, and Spark to build streaming services

Hive, Presto, Spark, Druid, Jupyter, Airflow, Kafka, Flink, Kubernetes… crazy names for open source technologies that you would not find from the monolithic technology vendors of yesteryear.

Lyft and other modern digital companies understand that their technology architecture should serve two basic purposes:

  1. Facilitate the capture, refinement, curation, sharing, management, governance, and analysis of the company’s invaluable data assets
  1. Build a technology architecture that doesn’t get in the way of point #1

The economics of the modern business

Data is the economic asset of lasting and differentiated value. Data is the source of customer, product, and operational insights that the modern company uses to differentiate its products and services while driving towards operational excellence:

  • Reducing unplanned downtime
  • Reducing procurement, logistics, inventory, manufacturing, and distribution costs
  • Increasing customer acquisition, retention, maturation, and advocacy

And data science is the heart of the data-value creation process.

These data and analytics-centric companies are tightly integrating the data science and business teams in order to define the parameters of analytics success and to identify, capture, and operationalize the sources of customer, product, and operational value creation.

These organizations are mastering the integration of the DataOps, data science, and DevOps disciplines to fuel their data monetization value chain.

DataOps, data science, DevOps – all focused on accelerating the monetization of an asset that The Economist magazine declared “The world’s most valuable resource.” See “Data Curation: Weaving Raw Data into Business Gold” for more on monetizing the world’s most valuable resource.

Disposable technology summary

What lessons can we take away from modern digital companies?

  • Lesson #1: Focus on aligning the organization on identifying, capturing, and operationalizing new sources of customer, product, and operational value buried in the company’s data.
  • Lesson #2: Don’t implement a rigid technology architecture that interferes with Lesson #1.

These modern digital companies, through their aggressive open source architecture strategies, realize that they are not in the technology architecture business; they are in the data monetization and commercialization business.

Now, let’s take that stupid armor off Steph Curry and let it rain three-pointers!

And please listen to the replay of our “Pathways to the Intelligent Enterprise” Webinar, featuring Phil Carter, chief analyst at IDC, and SAP’s Dan Kearnan and Ginger Gatling.

This article originally appeared on LinkedIn and is republished by permission. Hitachi Vantara is an SAP global technology partner.


Bill Schmarzo

About Bill Schmarzo

Bill Schmarzo is CTO, IoT and Analytics at Hitachi Vantara. Bill drives Hitachi Vantara’s “co-creation” efforts with select customers to leverage IoT and analytics to power digital business transformations. Bill is an avid blogger and frequent speaker on the application of big data and advanced analytics to drive an organization’s key business initiatives. Bill authored a series of articles on analytic applications, and is on the faculty of TDWI teaching a course on "Thinking Like A Data Scientist." Bill is the author of “Big Data: Understanding How Data Powers Big Business” and "Big Data MBA: Driving Business Strategies with Data Science." Bill is also an Executive Fellow at the University of San Francisco School of Management, and Honorary Professor at NUI Galway at NUI Galway J.E. Cairnes School of Business & Economics.