Why Today’s Digital Businesses Need Anomaly Detection

Debbie Fletcher

It’s one thing to amass enormous amounts of data and visualize it through today’s business intelligence standby, the analytics dashboard. But it can be quite another thing to glean real-time, actionable insights from it, given the volume, velocity, and complexity of the data flowing to the dashboard.

If our ability to collect and store Big Data is essentially without limit, the same can’t be said for our capacity to consume and comprehend it, mining hundreds, thousands, or even millions of critical KPIs, and the complex relationships among them, for the insights they hold. In practice, both human limitations and the limitations of typical BI solutions can—and often do—result in missed opportunities for digital businesses to discern patterns, identify anomalies and their causes, and take corrective actions, all at the time unexpected business incidents occur.

The early promise of BI innovations has fallen short of meeting that objective, but new solutions are emerging to fulfill the promise.

Anomaly detection is the new standard for business intelligence

Things go wrong. And it is one primary function of the massive volume of metrics businesses compile to not only sound an alarm when those incidents occur, but also to point the way to the cause and the solution. That’s where anomaly detection becomes an imperative.

In simplest terms, data anomaly detection uses algorithmic machine-learning methods to identify aberrations or even minor blips in expected patterns in a virtually unlimited number of metrics in real time. It is a type of artificial intelligence in which the anomaly detection program can, in effect, learn from its own “experience,” assimilating new data and changing accordingly. Practically speaking, that means that an anomaly detection system that can detect, say, several hundred types of business incidents today, as well as the interactions among the metrics contributing to the incidents, its knowledge base will grow, and so will its capacity to detect even more incidents in the future.

AI in its broadest sense is traditionally grounded in one of two techniques: supervised or unsupervised learning. In supervised learning, the system is seeded with classifications that can be applied to incidents it detects, provided the incidents it encounters are well-defined and fall appropriately into the classifications a programmer envisioned. In unsupervised learning, the system figures out for itself, so to speak, what incidents are normal and which are anomalies. The complication there is that the definition of “normal” can be fluid in the face of countless variables, where anomaly detection requires it to be precisely defined.

Given both the advantages and the drawbacks of each machine-learning approach, state-of-the-art anomaly detection systems have hybridized them into a semi-supervised learning technique better able to respond to the complexities of anomaly detection.

Similarly, cutting-edge systems hybridize the two traditional methods of detecting anomalies: the univariate and multivariate models. Univariate, as the term implies, examines each discrete metric, tracking normal patterns and identifying anomalies. It is straightforward and useful for exposing many incidents, but it leaves it to a marketer to analyze how anomalies in multiple KPIs relate to and influence one another, for just one example. Multivariate detection, in contrast, looks at all of the metrics at once to produce a single output that might reveal a problem but without pinpointing the metric at the root of the issue. The hybrid approach, as with machine learning, combines the best features of each, looking for anomalies at the single metric level and then grouping the related metrics in order to provide a comprehensive interpretation of the data.

There are a million reasons a business should adopt such a solution. Literally.

The volume of data any digital business consumes, coupled with the intricate interactions between and among KPIs, can be impenetrable for even the best marketers and the best BI dashboards, especially when an incident can’t wait through days or weeks of analysis for correction. Automated, real-time detection can uncover important insights in even the most obscure and easily overlooked corners of any data set.

By contrast, businesses operating without real-time automated anomaly detection, typically rely on dashboards to reveal issues and insights contained in the data.

On the one hand, that may mean that marketers pore over reports after the fact to discover patterns and scout for opportunities. In either case, they are necessarily working with a modest number of metrics and likely to miss many subtle but potentially costly anomalies. At the same time, they are working retrospectively and losing the impact of real-time insights. Alternatively, some businesses attempt to identify anomalies by setting thresholds on KPIs to trigger alerts when the numbers go either too high or too low, a difficult and overly objective task when compared to the efficacy of machine-learning systems.

Machine learning is changing all the rules

As revolutionary as dashboards seemed to be when they came on the scene, the sobering fact is they haven’t lived up to the hopes businesses once attached to them. They are valuable tools for management reporting, certainly, and for tracking a finite range of metrics, but artificial intelligence solutions have rapidly overtaken them when it comes to many management processes, including data anomaly detection.

Today’s machine-learning systems are moving businesses squarely into the future, allowing them to leverage data more efficiently and effectively. Machine-learning algorithms, by definition, grow more intelligent as they gain experience, outpacing all other solutions, turning even outlying data into actionable insights, and significantly enhancing the quality of the BI readily available to companies.

For more on how AI and machine learning can boost security, see Machine Learning—One New Weapon To Combat Fraud.

About Debbie Fletcher

Debbie Fletcher is an enthusiastic, experienced writer who has written for a range of different magazines and news publications over the years. Graduating from City University London specializing in English Literature, Debbie has a passion for writing has since grown. She loves anything and everything technology, and exploring different cultures across the world. She's currently looking towards starting her Masters in Comparative Literature in the next few years.