Using Exploratory Analytics To Find That Needle In The Data Haystack

Pierpaolo Vezzosi

Recently a performance artist at a modern art museum in Paris searched an entire haystack for a single actual needle. Stem by stem and leaf by leaf, artist Sven Sachsalber sorted through the hay. After two exhausting days, he actually found the needle. That’s the kind of performance that some of us face daily in our businesses.

How many of us analyze a new data set, wondering if we would ever find the insight we seek? How long might it take to find insight? Is the information even in there? Could the data set contain some unexpected, potentially more valuable information?

Businesses have long used classic analytic tools to manually sort through giant stacks of data, just like the performance artist. Fortunately, new analytic functionality deliver insight faster and with less effort. This functionality, called exploratory analytics, can quickly and automatically point business users to the most useful information hidden in a data set. How do traditional classic and advanced analytics solutions compare with exploratory analytics?

Understanding classic and advanced analytics

For the past 30 years, companies have relied on classic analytics solutions to create business intelligence. With these tools, analysts use a manual trial-and-error approach to find information within a data set. They create and test hypotheses, run queries, retrieve data, and filter the results, hoping for useful information to surface. Output of this analysis is presented in reports or dashboards, which analysts can share with their community. Results are not usually actionable, but they can be used to support or influence decisions.

In contrast, advanced analytics uses mathematical algorithms and computer automation to find patterns in data. These patterns are used to support decision making in operational environments. A data scientist sets up algorithms to explore regression, classification, clustering, association, time-series analysis, and outlier detection. These algorithms help the user understand the internal structure of an existing data set, and they allow the extrapolation of rules that can be applied to new data.

Rules, also known as models, can be embedded into applications and decision making processes to support operational users in their daily tasks. Advanced analytics can help businesses answer forward-looking questions, such as “What is the likelihood the customer will purchase this product?” They can also suggest next-best actions by answering questions such as “How many items should I have in stock next Tuesday?”

Gaining better insight with exploratory analytics

Exploratory analytics uses the methods and technologies of advanced analytics to help business analysts find useful insights in data sets. The solutions use data mining algorithms to automatically highlight the most important findings and suggest the best way to visualize the information from a select data set. The figure shows how exploratory analytics solutions use automated, advanced algorithms and visualization techniques to move beyond classic analytics.

Exploratory Analytics.4.png

With exploratory analytics, business analysts can find information about key influencers of measures, outliers, anomalies, points of interest, hidden structures (such as associations between values), and groups of records showing similarities. These solutions can also propose visualizations that help analysts to better grasp the meaning of the data in the data set, its relevance, and its value to the business problem.

Using exploratory analytics, business analysts can better understand the content of the data set and more effectively judge its business value. This initial set of pre-analyzed information helps analysts concentrate on the most important parts of the data. Essentially, exploratory analytics reduces the noise inherent in huge volumes of data. By providing a smaller but more useful collection of information than traditional analytics techniques, exploratory analytics can help businesses gain insight that can improve operations and results. Are there areas of your business that would benefit from exploratory analytics?

Want more on exploratory analytics? See Part One: The Rise of Exploratory Analytics.

Pierpaolo Vezzosi

About Pierpaolo Vezzosi

Pierpaolo Vezzosi is the Director of Product Management at SAP, responsible for the vision, strategy and outbound product management for the SAP Analytics offering in the areas of Predictive Analytics, big data and SAP BI offering on SAP HANA.