Why Today’s Digital Businesses Need Anomaly Detection

Debbie Fletcher

It’s one thing to amass enormous amounts of data and visualize it through today’s business intelligence standby, the analytics dashboard. But it can be quite another thing to glean real-time, actionable insights from it, given the volume, velocity, and complexity of the data flowing to the dashboard.

If our ability to collect and store Big Data is essentially without limit, the same can’t be said for our capacity to consume and comprehend it, mining hundreds, thousands, or even millions of critical KPIs, and the complex relationships among them, for the insights they hold. In practice, both human limitations and the limitations of typical BI solutions can—and often do—result in missed opportunities for digital businesses to discern patterns, identify anomalies and their causes, and take corrective actions, all at the time unexpected business incidents occur.

The early promise of BI innovations has fallen short of meeting that objective, but new solutions are emerging to fulfill the promise.

Anomaly detection is the new standard for business intelligence

Things go wrong. And it is one primary function of the massive volume of metrics businesses compile to not only sound an alarm when those incidents occur, but also to point the way to the cause and the solution. That’s where anomaly detection becomes an imperative.

In simplest terms, data anomaly detection uses algorithmic machine-learning methods to identify aberrations or even minor blips in expected patterns in a virtually unlimited number of metrics in real time. It is a type of artificial intelligence in which the anomaly detection program can, in effect, learn from its own “experience,” assimilating new data and changing accordingly. Practically speaking, that means that an anomaly detection system that can detect, say, several hundred types of business incidents today, as well as the interactions among the metrics contributing to the incidents, its knowledge base will grow, and so will its capacity to detect even more incidents in the future.

AI in its broadest sense is traditionally grounded in one of two techniques: supervised or unsupervised learning. In supervised learning, the system is seeded with classifications that can be applied to incidents it detects, provided the incidents it encounters are well-defined and fall appropriately into the classifications a programmer envisioned. In unsupervised learning, the system figures out for itself, so to speak, what incidents are normal and which are anomalies. The complication there is that the definition of “normal” can be fluid in the face of countless variables, where anomaly detection requires it to be precisely defined.

Given both the advantages and the drawbacks of each machine-learning approach, state-of-the-art anomaly detection systems have hybridized them into a semi-supervised learning technique better able to respond to the complexities of anomaly detection.

Similarly, cutting-edge systems hybridize the two traditional methods of detecting anomalies: the univariate and multivariate models. Univariate, as the term implies, examines each discrete metric, tracking normal patterns and identifying anomalies. It is straightforward and useful for exposing many incidents, but it leaves it to a marketer to analyze how anomalies in multiple KPIs relate to and influence one another, for just one example. Multivariate detection, in contrast, looks at all of the metrics at once to produce a single output that might reveal a problem but without pinpointing the metric at the root of the issue. The hybrid approach, as with machine learning, combines the best features of each, looking for anomalies at the single metric level and then grouping the related metrics in order to provide a comprehensive interpretation of the data.

There are a million reasons a business should adopt such a solution. Literally.

The volume of data any digital business consumes, coupled with the intricate interactions between and among KPIs, can be impenetrable for even the best marketers and the best BI dashboards, especially when an incident can’t wait through days or weeks of analysis for correction. Automated, real-time detection can uncover important insights in even the most obscure and easily overlooked corners of any data set.

By contrast, businesses operating without real-time automated anomaly detection, typically rely on dashboards to reveal issues and insights contained in the data.

On the one hand, that may mean that marketers pore over reports after the fact to discover patterns and scout for opportunities. In either case, they are necessarily working with a modest number of metrics and likely to miss many subtle but potentially costly anomalies. At the same time, they are working retrospectively and losing the impact of real-time insights. Alternatively, some businesses attempt to identify anomalies by setting thresholds on KPIs to trigger alerts when the numbers go either too high or too low, a difficult and overly objective task when compared to the efficacy of machine-learning systems.

Machine learning is changing all the rules

As revolutionary as dashboards seemed to be when they came on the scene, the sobering fact is they haven’t lived up to the hopes businesses once attached to them. They are valuable tools for management reporting, certainly, and for tracking a finite range of metrics, but artificial intelligence solutions have rapidly overtaken them when it comes to many management processes, including data anomaly detection.

Today’s machine-learning systems are moving businesses squarely into the future, allowing them to leverage data more efficiently and effectively. Machine-learning algorithms, by definition, grow more intelligent as they gain experience, outpacing all other solutions, turning even outlying data into actionable insights, and significantly enhancing the quality of the BI readily available to companies.

For more on how AI and machine learning can boost security, see Machine Learning—One New Weapon To Combat Fraud.

Comments

About Debbie Fletcher

Debbie Fletcher is an enthusiastic, experienced writer who has written for a range of different magazines and news publications over the years. Graduating from City University London specializing in English Literature, Debbie has a passion for writing has since grown. She loves anything and everything technology, and exploring different cultures across the world. She's currently looking towards starting her Masters in Comparative Literature in the next few years.

Machine Learning With Heart: How Sentiment Analysis Can Help Your Customers

Lance Hughes

When you think of artificial intelligence (AI), the word “emotion” doesn’t typically come to mind. But there’s an entire field of research using AI to understand emotional responses to news, product experiences, movies, restaurants, and more. It’s known as sentiment analysis, or emotion AI, and it involves analyzing views – positive, negative, or neutral – from written text to understand and gauge reactions.

Sentiment analysis can be used for survey research, social media analyses, and tracking psychological trends. Picture software that scans articles, reviews, ratings, and social media posts to determine sentiment changes for hotel guests. Hoteliers will, for example, aggregate and assess ratings and reviews in effort to improve guest satisfaction.

The tech behind sentiment analysis involves natural language processing or linguistic algorithms that assign values to positive, negative, or neutral text (converting opinions into datasets), while machine learning processes the datasets to reveal relevant trends over time. There’s significant planning required: How do you ensure the algorithms capture useful information? Are you identifying the right phrases to analyze? How can you convert findings into better products, services, and experiences?

At Concur, for instance, sentiment analysis has provided invaluable insights. Recently, Concur Labs and Concur UX Analytics developed a sentiment analysis tool for user product reviews. This tool automatically extracts themes to determine how customers feel about Concur’s service and helps identify which features people like most and which ones they find frustrating.

Emotion gauging is complicated

If we could categorize responses with just one emoji, that would easy. But humans are far more complicated and fascinating. This complexity applies to sentiment analysis. For example, comments like: “The film was very good,” are easy to analyze. But it gets a little harder when you add negation: “The film wasn’t bad.” It gets much harder when you add terms that would normally come across as positive but are actually negative based on context. For instance, “I wish this film was good. There were great many things it could have done right but didn’t.”

As a relatively new field, approaches are varied and maturing. Analysis has been traditionally conducted by taking what’s called a “bag of words” approach. Basically creating a list of all the words used along with how many times they were used. With this method, word order is thrown out the window. So “not bad” would come out as negative. Modern methods use recurrent neural networks called LSTMs (long short-term memory) to compress the entire sentence into a vector (a list of numbers) that encapsulates the meaning of the sentence, taking word order into account. This tends to have higher accuracy.

For businesses invested in customers, analyzing each piece of feedback by hand can be overwhelming. Sentiment analysis, developed within context, can help catch issues early and provide guidance on how to improve services. The related machine learning algorithms can take vast amounts of data; learn and perform specific tasks quickly; and sift through data based on your priorities. As the technology advances, businesses can benefit from these in-depth insights and customer satisfaction will surely follow suit.

Learn more about marketing in an increasingly data-driven era. Read about Influencing Customers Through Infinite Personalization.

Comments

Lance Hughes

About Lance Hughes

Lance Hughes is a principal creative technologist for Concur Labs. With a background in machine learning and mobile development, he helped design and develop multiple top-selling apps while working at Smashing Ideas and Sweet Action Games, a company he founded. When he's out of the office, Lance enjoys composing music, hosting deep learning meetups, spending time with his family, and exploring augmented and virtual reality.

Digitalist Flash Briefing: Answers To Two Burning Questions About Conversational AI

Bonnie D. Graham

Today’s briefing looks at a current hot topic – conversational AI and digital assistants for your business – from the perspective of another hot innovation from back in the day.

  • Amazon Echo or Dot: Enable the “Digitalist” flash briefing skill, and ask Alexa to “play my flash briefings” on every business day.
  • Alexa on a mobile device:
    • Download the Amazon Alexa app: Select Skills, and search “Digitalist.” Then, select Digitalist, and click on the Enable button.
    • Download the Amazon app: Click on the microphone icon and say “Play my flash briefing.”

Find and listen to previous Flash Briefings on Digitalistmag.com.

Read more on today’s topic

 

Comments

About Bonnie D. Graham

Bonnie D. Graham is the creator, producer and host/moderator of 29 Game-Changers Radio series presented by SAP, bringing technology and business strategy thought leadership panel discussions to a global audience via the Business Channel on World Talk Radio. A broadcast journalist with nearly 20 years in media production and hosting, Bonnie has held marketing communications management roles in the business software, financial services, and real estate industries. She calls SAP Radio her "dream job". Listen to Coffee Break with Game-Changers.

Diving Deep Into Digital Experiences

Kai Goerlich

 

Google Cardboard VR goggles cost US$8
By 2019, immersive solutions
will be adopted in 20% of enterprise businesses
By 2025, the market for immersive hardware and software technology could be $182 billion
In 2017, Lowe’s launched
Holoroom How To VR DIY clinics

Link to Sources


From Dipping a Toe to Fully Immersed

The first wave of virtual reality (VR) and augmented reality (AR) is here,

using smartphones, glasses, and goggles to place us in the middle of 360-degree digital environments or overlay digital artifacts on the physical world. Prototypes, pilot projects, and first movers have already emerged:

  • Guiding warehouse pickers, cargo loaders, and truck drivers with AR
  • Overlaying constantly updated blueprints, measurements, and other construction data on building sites in real time with AR
  • Building 3D machine prototypes in VR for virtual testing and maintenance planning
  • Exhibiting new appliances and fixtures in a VR mockup of the customer’s home
  • Teaching medicine with AR tools that overlay diagnostics and instructions on patients’ bodies

A Vast Sea of Possibilities

Immersive technologies leapt forward in spring 2017 with the introduction of three new products:

  • Nvidia’s Project Holodeck, which generates shared photorealistic VR environments
  • A cloud-based platform for industrial AR from Lenovo New Vision AR and Wikitude
  • A workspace and headset from Meta that lets users use their hands to interact with AR artifacts

The Truly Digital Workplace

New immersive experiences won’t simply be new tools for existing tasks. They promise to create entirely new ways of working.

VR avatars that look and sound like their owners will soon be able to meet in realistic virtual meeting spaces without requiring users to leave their desks or even their homes. With enough computing power and a smart-enough AI, we could soon let VR avatars act as our proxies while we’re doing other things—and (theoretically) do it well enough that no one can tell the difference.

We’ll need a way to signal when an avatar is being human driven in real time, when it’s on autopilot, and when it’s owned by a bot.


What Is Immersion?

A completely immersive experience that’s indistinguishable from real life is impossible given the current constraints on power, throughput, and battery life.

To make current digital experiences more convincing, we’ll need interactive sensors in objects and materials, more powerful infrastructure to create realistic images, and smarter interfaces to interpret and interact with data.

When everything around us is intelligent and interactive, every environment could have an AR overlay or VR presence, with use cases ranging from gaming to firefighting.

We could see a backlash touting the superiority of the unmediated physical world—but multisensory immersive experiences that we can navigate in 360-degree space will change what we consider “real.”


Download the executive brief Diving Deep Into Digital Experiences.


Read the full article Swimming in the Immersive Digital Experience.

Comments

Kai Goerlich

About Kai Goerlich

Kai Goerlich is the Chief Futurist at SAP Innovation Center network His specialties include Competitive Intelligence, Market Intelligence, Corporate Foresight, Trends, Futuring and ideation. Share your thoughts with Kai on Twitter @KaiGoe.heif Futu

Tags:

Blockchain: Much Ado About Nothing? How Very Wrong!

Juergen Roehricht

Let me start with a quote from McKinsey, that in my view hits the nail right on the head:

“No matter what the context, there’s a strong possibility that blockchain will affect your business. The very big question is when.”

Now, in the industries that I cover in my role as general manager and innovation lead for travel and transportation/cargo, engineering, construction and operations, professional services, and media, I engage with many different digital leaders on a regular basis. We are having visionary conversations about the impact of digital technologies and digital transformation on business models and business processes and the way companies address them. Many topics are at different stages of the hype cycle, but the one that definitely stands out is blockchain as a new enabling technology in the enterprise space.

Just a few weeks ago, a customer said to me: “My board is all about blockchain, but I don’t get what the excitement is about – isn’t this just about Bitcoin and a cryptocurrency?”

I can totally understand his confusion. I’ve been talking to many blockchain experts who know that it will have a big impact on many industries and the related business communities. But even they are uncertain about the where, how, and when, and about the strategy on how to deal with it. The reason is that we often look at it from a technology point of view. This is a common mistake, as the starting point should be the business problem and the business issue or process that you want to solve or create.

In my many interactions with Torsten Zube, vice president and blockchain lead at the SAP Innovation Center Network (ICN) in Potsdam, Germany, he has made it very clear that it’s mandatory to “start by identifying the real business problem and then … figure out how blockchain can add value.” This is the right approach.

What we really need to do is provide guidance for our customers to enable them to bring this into the context of their business in order to understand and define valuable use cases for blockchain. We need to use design thinking or other creative strategies to identify the relevant fields for a particular company. We must work with our customers and review their processes and business models to determine which key blockchain aspects, such as provenance and trust, are crucial elements in their industry. This way, we can identify use cases in which blockchain will benefit their business and make their company more successful.

My highly regarded colleague Ulrich Scholl, who is responsible for externalizing the latest industry innovations, especially blockchain, in our SAP Industries organization, recently said: “These kinds of use cases are often not evident, as blockchain capabilities sometimes provide minor but crucial elements when used in combination with other enabling technologies such as IoT and machine learning.” In one recent and very interesting customer case from the autonomous province of South Tyrol, Italy, blockchain was one of various cloud platform services required to make this scenario happen.

How to identify “blockchainable” processes and business topics (value drivers)

To understand the true value and impact of blockchain, we need to keep in mind that a verified transaction can involve any kind of digital asset such as cryptocurrency, contracts, and records (for instance, assets can be tangible equipment or digital media). While blockchain can be used for many different scenarios, some don’t need blockchain technology because they could be handled by a simple ledger, managed and owned by the company, or have such a large volume of data that a distributed ledger cannot support it. Blockchain would not the right solution for these scenarios.

Here are some common factors that can help identify potential blockchain use cases:

  • Multiparty collaboration: Are many different parties, and not just one, involved in the process or scenario, but one party dominates everything? For example, a company with many parties in the ecosystem that are all connected to it but not in a network or more decentralized structure.
  • Process optimization: Will blockchain massively improve a process that today is performed manually, involves multiple parties, needs to be digitized, and is very cumbersome to manage or be part of?
  • Transparency and auditability: Is it important to offer each party transparency (e.g., on the origin, delivery, geolocation, and hand-overs) and auditable steps? (e.g., How can I be sure that the wine in my bottle really is from Bordeaux?)
  • Risk and fraud minimization: Does it help (or is there a need) to minimize risk and fraud for each party, or at least for most of them in the chain? (e.g., A company might want to know if its goods have suffered any shocks in transit or whether the predefined route was not followed.)

Connecting blockchain with the Internet of Things

This is where blockchain’s value can be increased and automated. Just think about a blockchain that is not just maintained or simply added by a human, but automatically acquires different signals from sensors, such as geolocation, temperature, shock, usage hours, alerts, etc. One that knows when a payment or any kind of money transfer has been made, a delivery has been received or arrived at its destination, or a digital asset has been downloaded from the Internet. The relevant automated actions or signals are then recorded in the distributed ledger/blockchain.

Of course, given the massive amount of data that is created by those sensors, automated signals, and data streams, it is imperative that only the very few pieces of data coming from a signal that are relevant for a specific business process or transaction be stored in a blockchain. By recording non-relevant data in a blockchain, we would soon hit data size and performance issues.

Ideas to ignite thinking in specific industries

  • The digital, “blockchained” physical asset (asset lifecycle management): No matter whether you build, use, or maintain an asset, such as a machine, a piece of equipment, a turbine, or a whole aircraft, a blockchain transaction (genesis block) can be created when the asset is created. The blockchain will contain all the contracts and information for the asset as a whole and its parts. In this scenario, an entry is made in the blockchain every time an asset is: sold; maintained by the producer or owner’s maintenance team; audited by a third-party auditor; has malfunctioning parts; sends or receives information from sensors; meets specific thresholds; has spare parts built in; requires a change to the purpose or the capability of the assets due to age or usage duration; receives (or doesn’t receive) payments; etc.
  • The delivery chain, bill of lading: In today’s world, shipping freight from A to B involves lots of manual steps. For example, a carrier receives a booking from a shipper or forwarder, confirms it, and, before the document cut-off time, receives the shipping instructions describing the content and how the master bill of lading should be created. The carrier creates the original bill of lading and hands it over to the ordering party (the current owner of the cargo). Today, that original paper-based bill of lading is required for the freight (the container) to be picked up at the destination (the port of discharge). Imagine if we could do this as a blockchain transaction and by forwarding a PDF by email. There would be one transaction at the beginning, when the shipping carrier creates the bill of lading. Then there would be look-ups, e.g., by the import and release processing clerk of the shipper at the port of discharge and the new owner of the cargo at the destination. Then another transaction could document that the container had been handed over.

The future

I personally believe in the massive transformative power of blockchain, even though we are just at the very beginning. This transformation will be achieved by looking at larger networks with many participants that all have a nearly equal part in a process. Today, many blockchain ideas still have a more centralistic approach, in which one company has a more prominent role than the (many) others and often is “managing” this blockchain/distributed ledger-supported process/approach.

But think about the delivery scenario today, where goods are shipped from one door or company to another door or company, across many parties in the delivery chain: from the shipper/producer via the third-party logistics service provider and/or freight forwarder; to the companies doing the actual transport, like vessels, trucks, aircraft, trains, cars, ferries, and so on; to the final destination/receiver. And all of this happens across many countries, many borders, many handovers, customs, etc., and involves a lot of paperwork, across all constituents.

“Blockchaining” this will be truly transformational. But it will need all constituents in the process or network to participate, even if they have different interests, and to agree on basic principles and an approach.

As Torsten Zube put it, I am not a “blockchain extremist” nor a denier that believes this is just a hype, but a realist open to embracing a new technology in order to change our processes for our collective benefit.

Turn insight into action, make better decisions, and transform your business. Learn how.

Comments

Juergen Roehricht

About Juergen Roehricht

Juergen Roehricht is General Manager of Services Industries and Innovation Lead of the Middle and Eastern Europe region for SAP. The industries he covers include travel and transportation; professional services; media; and engineering, construction and operations. Besides managing the business in those segments, Juergen is focused on supporting innovation and digital transformation strategies of SAP customers. With more than 20 years of experience in IT, he stays up to date on the leading edge of innovation, pioneering and bringing new technologies to market and providing thought leadership. He has published several articles and books, including Collaborative Business and The Multi-Channel Company.