#GartnerBI and Big Data: Fear, Loathing, and Business Breakthroughs

Timo Elliott

big data

At the Gartner Business Intelligence and Analytics Summit in Barcelona, there was near universal agreement about three things to do with “big data”

  1. It’s an awful term (but we’re stuck with it)
  2. Whatever it means, it’s a big deal, and requires big changes to traditional information infrastructures
  3. It will result in big new business opportunities

“Big Data” is a terrible term

Gartner analyst Doug Laney first coined the term “big data” over over 12 years ago (at least in its current form – people have been complaining about “information overload” since Roman times). But the term’s meaning is still far from clear and it was nominated the #1 “tech buzzword that everyone uses but don’t quite understand” (followed closely by “cloud”).

When using the term, Gartner usually keeps the quote marks in place (i.e. it’s “big data”, not big data). Here’s the definition provided by analyst Ted Friedman to “de-hype” the term during the summit keynote:

“Big Data” are high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision-making”

Analyst Donald Feinberg warned people that “talking only about big data can lead to self-delusion” and urged people not to “surrender to the hype-ocracy.” He left left no doubt over where he stood on the use of the term: “Big data doesn’t mean MapReduce or Hadoop. Big data doesn’t exist, it’s meaningless, it’s ridiculous…” The audience started applauding, to which he replied: “Why are you clapping?! Why do you all fall for it? Why do the vendors do it?!

As SAP’s Jason Rose? put it “how can we demystify this? …easy, drop the ‘big’. Data has always been the key challenge in BI”.

For what it’s worth, here’s my tongue-in-cheek definition, and it’s worth noting that big data is quickly becoming the default term for what we used to call analytics or business intelligence.

But Big Data is a Big Deal

Despite the problems, Doug Laney noted that big data the most-searched-for term on  Why is it so popular? Maybe because it’s so nebulous that people want to check if they have understood it. Or maybe because there’s no other more precise term to indicate the new analytic opportunities. And maybe because “hype has a value” as Ted Friedman put it: big data has proved to be a new opportunity to talk to business people about the power of analytics, and because everybody’s searching for it, vendors would be crazy not to include it in their marketing.

Conference attendees generally believed that the biggest opportunity for big data analysis was new insights from “dark data” that lies unused within organizations today. Gartner highlighted the dangers of implementing shiny new big data technology, separate from existing analytics infrastructures: “Do not make your big data implementations siloed. Make them part of the overall strategy for BI.” said Ted Friedman. “Link to stuff you are already doing. Don’t make big data a standalone thing. And don’t feel like you’ve got to go out and buy a whole new technology stack.”

Analyst Rita Sallam, in a session on data variety, gave some examples of the new opportunities:

Some of Gartner’s public predictions related to big data:

  • By 2015, 65 percent of packaged analytic applications with advanced analytics will come embedded with Hadoop.
  • By 2016, 70 percent of leading BI vendors will have incorporated natural-language and spoken-word capabilities.
  • By 2015, more than 30 percent of analytics projects will deliver insights based on structured and unstructured data.

But What Does It Mean For The Business?

Donald Fienberg: “Realize that big data is not about doing ‘more’ of the same thing – it’s about doing things differently”. “The major opportunities for big data are around ways to transform the business and disrupt the industry” said Doug Laney. These included radically changing existing business processes, introducing new, more-personalized products and services, and “answering chewy questions that weren’t possible before.” Some examples:

  • NetFlix did deep analysis of their viewers’ preferences, and used that to craft the new “House of Cards” TV series – a $100M investment
  • New financial lenders are using big data to find untapped banking opportunities – including lending scores based on what you say on social media
  • Passur uses big data to provides real-time monitoring of air traffic, to potentially save millions of dollars per year. Today, pilots’ estimated times of arrival are off by more than ten minutes ten percent of the time, and five minutes 30% of the time – knowing exactly when the planes will arrive means more automation, better operating efficiencies, improved security, etc.
  • Enologix analyzes the chemical composition of new wines to predict wine spectator score, and offer advice on how to improve the score
  • Dollar General, Kroger and other retailers provide data to partners to analyze, for “free strategic advice”
  • Insurance companies are using text mining on previously-unexamined “dark data” on claims forms to sniff out indicators of fraud
Mats-Olov Eriksson of, a Sweden-based casual gaming site (card games, social games, etc), gave an entertaining presentation called “Beyond Big and Data”, hosted by recently-returned Gartner analyst Frank Buytendijk. It was illustrated with more cute kittens, puppies, and otters than I’ve ever seen before in a technology conference!


Mat’s title is “Spiritual Leader, Data Warehouse”, and he explained how the company collects massive amounts of log data about user activities, then uses a combination of Hadoop, traditional data warehousing, and BI cubes to optimize the player experience – and the company’s profits.

The company works with Facebook on games like Candy Crush Saga. The company’s main revenue comes from in-app purchases: for example, you might get five free lives in a game, and if you run out you can either wait for a period, or pay a small sum to play immediately. To try to figure out what options generate the most revenue, and keep players in the game, the company currently processes 60,000 log files per day. The files contain around three billion “events” per day, and the company expects that to rise to six billion by the end of this year. Mats explained that big data is best compared to a software development project, requiring hard-to-find skilled personnel. He characterized hadoop as a “powerful but immature five year old” and explained that they used a MySQL data mart for low-latency access: “HIVE is anything but low latency – it takes fifteen seconds before you even get an error message!”.

One of the recent challenges the company has faced is information governance – a term they hadn’t even used until six months ago. Mats noted that the European privacy laws were by far the “harshest” they deal with.

In a separate presentation, that I unfortunately didn’t attend, Tom Fastner of eBay gave an overview of his company’s analytic systems, explaining that they process more than 100 Petabytes of data I/Os every day, and a “singularity” keyvalue store on Teradata manages 36 Petabytes of data. The largest table is 3 Petabytes / 3 trillion rows, and only takes 32 seconds to scan. Here’s a presentation from 2011 that talks about how eBay combines structured, semi-structured, and unstructured data.

The inimitable Steve Lucas says of big data: “when you see it, you know it”. MKI, a bioscience company is using big data to analyze genome data, using a combination of hadoop, open-source “R” algorithms, and SAP’s in-memory platform, SAP HANA (registration required).




Here’s a summary of the MKI project:

In Conclusion

Big data is a lousy term, but offers big opportunities in return for some big information infrastructure changes. What does the future hold? Let’s hope less of the hype, and more of the business change…



About Timo Elliott

Timo Elliott is an innovation evangelist and international conference speaker who has presented to business and IT audiences in over forty countries around the world. A 23-year veteran of SAP BusinessObjects, Elliott works closely with SAP development and innovation centers around the world on new technology directions. His popular Business Analytics blog at tracks innovation in analytics and social media, including topics such as big data, collaborative decision-making, and social analytics. Prior to Business Objects, Elliott was a computer consultant in Hong Kong and led analytics projects for Shell in New Zealand. He holds a first-class honors degree in Economics with Statistics from Bristol University, England.

Recommended for you:

13 Scary Statistics On Employee Engagement [INFOGRAPHIC]

Jacob Shriar

There is a serious problem with the way we work.

Most employees are disengaged and not passionate about the work they do. This is costing companies a ton of money in lost productivity, absenteeism, and turnover. It’s also harmful to employees, because they’re more stressed out than ever.

The thing that bothers me the most about it, is that it’s all so easy to fix. I can’t figure out why managers aren’t more proactive about this. Besides the human element of caring for our employees, it’s costing them money, so they should care more about fixing it. Something as simple as saying thank you to your employees can have a huge effect on their engagement, not to mention it’s good for your level of happiness.

The infographic that we put together has some pretty shocking statistics in it, but there are a few common themes. Employees feel overworked, overwhelmed, and they don’t like what they do. Companies are noticing it, with 75% of them saying they can’t attract the right talent, and 83% of them feeling that their employer brand isn’t compelling. Companies that want to fix this need to be smart, and patient. This doesn’t happen overnight, but like I mentioned, it’s easy to do. Being patient might be the hardest thing for companies, and I understand how frustrating it can be not to see results right away, but it’s important that you invest in this, because the ROI of employee engagement is huge.

Here are 4 simple (and free) things you can do to get that passion back into employees. These are all based on research from Deloitte.

1.  Encourage side projects

Employees feel overworked and underappreciated, so as leaders, we need to stop overloading them to the point where they can’t handle the workload. Let them explore their own passions and interests, and work on side projects. Ideally, they wouldn’t have to be related to the company, but if you’re worried about them wasting time, you can set that boundary that it has to be related to the company. What this does, is give them autonomy, and let them improve on their skills (mastery), two of the biggest motivators for work.

Employees feel overworked and underappreciated, so as leaders, we need to stop overloading them to the point where they can’t handle the workload.

2.  Encourage workers to engage with customers

At Wistia, a video hosting company, they make everyone in the company do customer support during their onboarding, and they often rotate people into customer support. When I asked Chris, their CEO, why they do this, he mentioned to me that it’s so every single person in the company understands how their customers are using their product. What pains they’re having, what they like about it, it gets everyone on the same page. It keeps all employees in the loop, and can really motivate you to work when you’re talking directly with customers.

3.  Encourage workers to work cross-functionally

Both Apple and Google have created common areas in their offices, specifically and strategically located, so that different workers that don’t normally interact with each other can have a chance to chat.

This isn’t a coincidence. It’s meant for that collaborative learning, and building those relationships with your colleagues.

4.  Encourage networking in their industry

This is similar to number 2 on the list, but it’s important for employees to grow and learn more about what they do. It helps them build that passion for their industry. It’s important to go to networking events, and encourage your employees to participate in these things. Websites like Eventbrite or Meetup have lots of great resources, and most of the events on there are free.

13 Disturbing Facts About Employee Engagement [Infographic]

What do you do to increase employee engagement? Let me know your thoughts in the comments!

Did you like today’s post? If so you’ll love our frequent newsletter! Sign up here and receive The Switch and Shift Change Playbook, by Shawn Murphy, as our thanks to you!

This infographic was crafted with love by Officevibe, the employee survey tool that helps companies improve their corporate wellness, and have a better organizational culture.


Recommended for you:

Supply Chain Fraud: The Threat from Within

Lindsey LaManna

Supply chain fraud – whether perpetrated by suppliers, subcontractors, employees, or some combination of those – can take many forms. Among the most common are:

  • Falsified labor
  • Inflated bills or expense accounts
  • Bribery and corruption
  • Phantom vendor accounts or invoices
  • Bid rigging
  • Grey markets (counterfeit or knockoff products)
  • Failure to meet specifications (resulting in substandard or dangerous goods)
  • Unauthorized disbursements

LSAP_Smart Supply Chains_graphics_briefook inside

Perhaps the most damaging sources of supply chain fraud are internal, especially collusion between an employee and a supplier. Such partnerships help fraudsters evade independent checks and other controls, enabling them to steal larger amounts. The median loss from fraud committed
by a single thief was US$80,000, according to the Association of Certified Fraud Examiners (ACFE).

Costs increase along with the number of perpetrators involved. Fraud involving two thieves had a median loss of US$200,000; fraud involving three people had a median loss of US$355,000; and fraud with four or more had a median loss of more than US$500,000, according to ACFE.

Build a culture to fight fraud

The most effective method to fight internal supply chain theft is to create a culture dedicated to fighting it. Here are a few ways to do it:

  • Make sure the board and C-level executives understand the critical nature of the supply chain and the risk of fraud throughout the procurement lifecycle.
  • Market the organization’s supply chain policies internally and among contractors.
  • Institute policies that prohibit conflicts of interest, and cross-check employee and supplier data to uncover potential conflicts.
  • Define the rules for accepting gifts from suppliers and insist that all gifts be documented.
  • Require two employees to sign off on any proposed changes to suppliers.
  • Watch for staff defections to suppliers, and pay close attention to any supplier that has recently poached an employee.

About Lindsey LaManna

Lindsey LaManna is Social and Reporting Manager for the Digitalist Magazine by SAP Global Marketing. Follow @LindseyLaManna on Twitter, on LinkedIn or Google+.


Recommended for you:

What If Chelsea Manager Jose Mourinho Could Be Proved Right In Medical Staff Row?

Mark Goad

Big Data and the Internet of Things brings new level of insight to sports medicine

With the 2015-16 European football (soccer) season underway, we are already seeing the impact of the huge pressure to succeed. In some cases, it is boiling over even this early on, with Chelsea manager Jose Mourinho getting involved in a very public row with his medical staff over the treatment of Eden Hazard during a match. As the season builds momentum, all clubs know one of the most vital aspects of winning trophies is keeping the best players fit so they can play at the top of their game as often as possible.

Last season, just like in every season, we saw injuries that affected teams’ results and possibly their final standings at the end of the season, while other teams capitalized. Arsenal manager Arsene Wenger blamed injuries for the team’s failed title bid, while Real Madrid suffered injuries to players like Gareth Bale and Luka Modric at a crucial stage of the season and lost the title to Barcelona.

There’s no doubt that football clubs, especially the bigger teams, employ first-rate medical staff – physiotherapists, doctors, sports scientists, and so on – but they can only do so much to keep players off the treatment table. Players are human, after all, and keeping them injury-free for such long and grueling campaigns is a big ask. This season again will see players on the end of crunching tackles, over-exerting their bodies, and over-stretching.

What’s less talked about than lost games and league titles when discussing injuries is the salaries paid to injured players. The estimated average cost of player injuries in the top four professional football leagues in 2015 was $12.4 million* per team. Remarkably, every year teams lose an equivalent of 15%-30%** of their player payroll to injuries.

As salaries continue to rise, injuries are becoming just as much of an off-the-pitch boardroom issue as they are an on-the-pitch issue. Consider that if Barcelona’s Lionel Messi, the world’s highest-paid player, spends just a week out injured, the club still has to pay his weekly salary of around $1 million. Not only that, but there’s the huge potential for lost revenue from missing out on UEFA Champions League progress or domestic success because key players are out.

Just as winning seems to mean more than ever, so does football as a business. So with the spotlight firmly on “sweating the assets” – extracting maximum value from the entire squad – clubs are looking to Big Data and Internet of Things technology to consider how player injuries can be prevented with new levels of insight.

Prevention is better than cure

In July this year we saw what could be a huge landmark in the potential of monitoring the risk of injuries, when football’s international governing body FIFA announced its approval of wearable electronic performance and tracking systems during matches. As well as collecting data on statistics like distance covered and heart rate to determine decisions like substitution timings, this also paves the way for wearable satellite devices that keep medical staff updated on the likelihood of a player picking up an injury from over-exertion.

Emerging injury-risk monitoring software uses the concepts of Big Data and wearable technology to pull in and apply mathematical formulas to an exhaustive range of relevant data about players: fitness levels, recent levels of exertion, opponents, age, technique, hydration, even weather. This could help medical staff predict the risk of future injuries with much greater accuracy, allowing them to run simulations and take corrective actions in real time. Imagine a seemingly non-injured key player being substituted during a tightly contested match, only to find out afterwards that monitoring software had indicated he was at a high risk of pulling a muscle. This could very much be a part of the future of professional football.

Going back to Jose Mourinho and his reaction to the Chelsea medical staff running onto the pitch to treat Eden Hazard, it’s interesting to consider how in the future this kind of technology could either support or discredit his position in the dispute. It could help managers work more closely with physiotherapists, as they can visualize the data that shows the risk of injury to players. Although the pressure to win will likely keep on rising, the risk of expensive players injuries could see a big reduction.

SAP’s own injury risk monitoring software is currently in the proof-of-concept phase and will be entering development in the near future. The goal is to build IRM on the SAP Sports One platform as an additional component, and to provide integration to the existing modules of SAP Sports One solution. SAP Sports One was launched earlier this year and is the first sports-specific cloud solution powered by the SAP HANA platform, providing a single, unified platform for team management and performance optimization.

*Statistic calulated using 2015 Global Sports Salaries Survey

**Bleacher Report “Inside the 2014 Numbers of Each MLB Team’s Regular-Season Injury Impact” and NBA Injury Analysis


Mark Goad

About Mark Goad

Mark Goad, Value Advisory Associate, SAP Canada, is an experienced business analyst with industry coverage spanning telecommunications & retail, with a focus on digital business models. He specializes in synthesizing industry trends with a detailed analysis of client-specific data to help customers build out high-impact business & IT strategies. Outside of work, Mark volunteers as a lead management consultant for Junior Achievement of Central Ontario and contributes to a range of thought leadership publications.


Recommended for you:

4 Biggest Risks In NOT Using Social Media

April Crichlow

These days social media is critical for success in business. Early adopters have made great strides, using it to engage with customers online and find new clients. For the laggards — typically small businesses that think they don’t have the resources or need for social media — the question looms: “Is social media a fad, or is it here to stay?”

Unfortunately for these companies, social media is here to stay. There are four major risks in not using social platforms as a business tool:

  1. You risk being out of the loop. Social media is a key channel for consumers collecting information and connecting with other consumers. It is also a great opportunity for companies to engage with current customers, as well as potential customers, all over the world. By not using social media, you run the risk of losing customers, credibility, and crucial information that can benefit your business. Even if you choose not to actively participate in discussions, you need be aware and stay informed regarding conversations about your company. Don’t stick your head in the ground and hope for social media to “blow on by.”
  1. You can’t respond to negative comments about your business. When customers are not satisfied with your product or service, one of the first things many will do is complain on Twitter or Facebook, or they will write a bad review online. If you are not actively keeping tabs on these discussions and reviews, they can hurt your reputation and cost you potential business. How can you protect your brand if you don’t know what’s being said about it online? Social media is now the default platform for customer service. Instead of calling an 800 number, consumers want to send businesses a tweet or post something on a Facebook page. When they can’t find you online, they will go to a review site such a Yelp or Merchant Circle to complain and warn other customers. However, if they have a relationship with your company, they are much less likely to take such actions and will instead send you an email or a private message about the problem.
  1. You risk missing the positive comments about your business. Customers also leave positive feedback online about companies with which they do business. However, if they believe their comments won’t be read by the companies they are praising, satisfied customers are less likely to leave feedback.
  1. You risk giving your competitors an unfair advantage. If your competitors are active on social media and you are not, your rivals have a leg up on winning business from potential customers. You don’t allow for comparisons and can’t answer questions in real time. Unless your product or service is overwhelmingly superior, this is one risk you cannot afford to take!

Social media is an excellent forum to participate in discussions happening right now about your business and your industry. Building an active presence on social sites offers numerous opportunities to promote your products and services, provide outlets for customer service, and check up on your competition. It’s not too late to start using social media as a business tool…but one day soon, it might be.

If you are an SAP partner and would like to learn more about this topic, join me on Dec 1st for How to Spend 15 Minutes a Day on Social Without Breaking a Sweat. Register now: (s-user) #SAPMarketingAcademy


Recommended for you: