When Analytics Fail

Nick Petri

Image provided by: ETF Prodigy

After hearing the redundancy of my title, Analyst on OpenView’s Research and Analytics team, most people naturally assume I love numbers. Often I tell them I do. Numbers boil down an infinitely complex world into a manageable format that can be manipulated, modeled, and mined for insight. Accurate numbers, properly used, can teach you a ton about how the world works, and in turn, the adjustments you make based on that analysis can affect the world in tangible and profound ways.

But numbers can’t answer everything, and if you follow them blindly, you can be badly misled.

Too many executives, whether “numbers people” or not, see statistical analysis as a necessary stamp of approval to any business decision.  But the simple truth is that there isn’t a mathematical answer to every question, and if you force one using data that doesn’t accurately describe the real world, no amount of color-coded charts, quadratic regressions, and chi-squared tests will yield an ounce of insight.

These situations are incredibly common in the business world. Unlike physicists, chemists, or biologists, business people are unable to run tightly controlled experiments. We can’t scour the earth for cool problems with clean statistical answers, like academics can. The problems come to us, and they’re usually messy and fraught with biases.

While the statistical discipline has very good tools for identifying relationships within a data set, it’s blind to the quality of the data itself. Here are three common problems that can render a statistically significant conclusion dead wrong:

  • Self-Selection: Survey or interview respondents often respond because they have good things to say about the subject. Likewise, firms prefer to disclose financial information when it’s favorable or improving. This type of problem paints a rosier picture, on average, than is really the case.
  • Missing Data: More information is almost always available on larger companies and more recent events. When there’s missing information, do you come up with a proxy, estimate the missing fields, or exclude those entries altogether? Any solution you choose will introduce a layer of noise into the equation.
  • Tenuous Proxies: Say we’re trying to measure the impact of marketing spend on customer acquisition for a particular industry. Since neither variable is public, we’ll have to use proxies and assumptions to estimate them. It’s important to remember that the ultimate conclusion isn’t measuring the actual variables, but their proxies: the strength of the conclusion relies heavily on how close the two are. Since we don’t have the actual variables, this can be very difficult to measure.

The result is a difficult balancing act: you have to tolerate moderate levels of bias within the data to reach a conclusion, but as the problems pile up, the validity of that conclusion declines no matter how statistically significant it may appear.

Still, an analyst is being paid to solve a problem, and to some, an inconclusive result is the worst kind of failure. It can be tempting to present a conclusion as a home run despite obvious flaws in the methodology. The best analysts properly communicate their level of confidence in the results, and know when to admit that the data at hand is inconclusive, even if it isn’t what their stakeholders were hoping to hear. Doing otherwise can be extremely destructive.

So do I love numbers? I’d say our relationship is rocky. I certainly respect them as a powerful tool to understand the world around me, but am realistic about their limitations. Statistical conclusions are only as good as the data that goes into them, and there isn’t good data for every problem.


Recommended for you:

Optimism and Alarm as Events Continue to Shake Market Confidence

Derek Klobucher

“The financial service industry has an image problem,” market information and strategies magazine Financial Advisor wrote after last week’s SIFMA Private Client Conference. “Earning back trust was a common theme during the one-day conference.”

There was optimism as well, with talk of nobility in financial services, increasing the “standard of care” and “stewardship.” But many acknowledged the damage done by market manipulation, bad actors and poor performance, with one speaker venturing that Generation X is completely soured on investing.

After tales of Goldman Sachs managing directors calling clients “muppets,” can you blame Generation X? Even recent activity would validate its concerns.

Shelved Confidence

A U.S. federal court recently fined Amsterdam-based high-frequency trading firm Optiver $14 million in relation to multiple attempts to manipulate oil prices with the disruptive “banging the close” trading technique out of its Chicago office in 2007. Three Optiver employees received two- to eight-year bans from trading commodities.

So it is no surprise that more than half of Americans do not trust bankers and brokers to do what is best for the economy. We could justifiably be surprised that the number is only as high as 54 percent. Fidelity Investments’ Kathleen Murphy shared that figure from a CNN poll at the SIFMA Conference, adding that the number is up from 30 percent in 1990.

But this isn’t just a matter of holding culprits accountable. It’s also about holding authorities accountable for punishing wrongdoers — and not-exactly-wrongdoers.

Market Capital Punishment

That initiative suffered a setback recently when U.K. regulators with the Financial Services Authority failed to fine former UBS executive John Pottage £100,000 for not acting fast enough to fix compliance issues within his arm of the firm. The unprecedented attempt to punish senior management for poor oversight could have been a watershed for restoring confidence in the financial services industry.

Questions aside about who defines “fast enough” and whether or not it is reasonable to make an example of someone in the name of restoring confidence, this case telegraphs regulatory intentions. Authorities will surely learn from this defeat, and make similar attempts with different tactics in the future.

And there is something to be said for visible effort.

Louder than Words

“High-profile crackdowns on insider trading seen in the U.S. and U.K. in the wake of the financial crisis have not been mirrored in Japan,” The Financial Times noted Sunday. “Examples of punishment for market abuses are rare.”

Financial services firms could put their clients before themselves, one speaker at the SIFMA conference suggested, even taking on the role of “a good parent” or caring physician.

But just one thing will improve confidence, according to Fidelity’s Murphy.

“Not by asking for it, not by advertising,” she said. “We need to earn their trust by actions.”


Recommended for you:

The Most Important Words In Social Media

Michael Brenner

I can no other answer make, but, thanks, and thanks.”  ~ William Shakespeare

And so I offer up to you that the most important words in social media are “thank you!”

They may be the most important words in, well, the history of words. But in social media, where we only see avatars and not real expressions on real human faces, they are particularly important.

I recently read Tweet Smarter Not Harder by Rachel Thompson (@RachelintheOC) and I was reminded about how simply saying “thank you” has helped me make amazing connections on the social channels where I interact with people, sometimes and often on a daily basis.

So I will explain my approach to be thankful on social media here in this post.

And whether you use Twitter, Facebook, Pinterest, Linkedin, Tumblr,  or any of the social tools available, remember that you are connecting with people, not avatars.  And that a little “thank you” goes along way!

What the heck does “Magnanimous” mean?

We make a living by what we get, but we make a life by what we give.”  ~ Winston Churchill

I met Bill Lublin (@BillLublin) at a social media conference here in Philadelphia. We were on Triberr together and had the pleasure of reviewing and sharing each other’s blog posts on a regular basis.

After I thanked him for his support and told him how great it was to meet him in real life (aka #irl), he introduced me to his colleague as the most “magnanimous man in social media.”

I didn’t really know what magnanimous meant (it’s OK! Look it up…I had to.) but I guessed from his context that he meant to say that I was a nice guy or something – which is pretty cool, right?

“Hey man, thanks a lot!” I said. And didn’t think much about it for weeks. Then a few months later, I was asked by Bill Strawderman (@marketingbard) to present on personal branding to a group of volunteer bloggers and I found myself using the word. I instructed the audience to be magnanimous.

So afterwards I looked it up. It basically means “generosity” but literally means “being of noble mind.” So while it maybe really hard to say (and not so easy to spell), I suggest we should all think about being noble and generous as you approach your social activity.

The Thank You Economy?

As we express our gratitude, we must never forget that the highest appreciation is not to utter words, but to live by them.” ~ JFK

Last year, famous social media influencer and entrepreneur Gary Vaynerchuk (@GaryVee) wrote The Thank You Economy (not an affiliate link). In the book, Gary talks about how the internet and social has – both consumers and business buyers – given our voices back and transferred the balance of economic power to each of us.

He talks about how it is not the company with the biggest budget who will be successful in the new business climate of social interactivity, but it is the company who cares the most about their customers and makes them feel like they have a personal connection with a brand that will succeed.

I have not read the book (yet – summer reading list), and I’m not endorsing it (yet) but I totally agree with him that “businesses [that] can harness all the changes and challenges inherent in social media [can] turn them into tremendous opportunities for profit and growth.”

Thank Everyone?

From the time I first started on Twitter (on July 9th, 2009) I thanked everyone who ReTweeted me. I follow back every real human being (as far as I can tell). I answer every @ mention and I respond to any direct message that doesn’t look like SPAM or an auto-reply.

Some people might think I am “crowding the twitter stream” with useless “thank you’s”. But for me, it’s the least I can do if you take the time to share something of mine or mention me. I also try to ReTweet and @ mention and comment on blog posts to return the favor. But I always thank my ReTweeters.

My Conclusion: Saying “thank you” and being “of noble mind” is important for personal success. It is especially important for personal social media success and it may be just as important to build gratitude into your business.

Looking To Get Started On Twitter

If you’re not already a Twitter rock star, go back and read Rachel’s post (and the first part of her series) linked above or read my early post on how to get started on Twitter.

Or check out my 10 Tips for Twitter Success. I mention “being nice” but maybe I should have said “be magnanimous!”

Photo Source


Recommended for you:

Introducing the Self-Service Generation

Michael Mischker

ClickZ discuss how the proliferation of mobile devices and the formation of a new breed of customer are driving a revolution in requirements for access to business data:

“Look in the mirror. We are living in the self-service data generation. All ages, from all walks of life, obsessively use the Internet and mobile devices for communication, life management, and purchase decision making. Users have an unprecedented sophistication in using data for personal decisions. Many have grown up with technology at our fingertips, and believe instant access to information is a right. It’s not surprising that consumers demand intelligent services and access to our own data from businesses and government agencies. Consumers want to “co-own” the data – no longer desiring or trusting marketers to make decisions about what is desirable to read and experience.

This trend in consumer behavior is creating a need for new “consumer intelligence” aspects of our business data warehousing and management – and putting pressure on the automation software we use. Consumer intelligence expands the field of business intelligence (BI), which has long been the key to understanding marketing data for automating and optimizing decisions about offer selection, placement, and cadence”.

Read the full article here.


Recommended for you:

Leverage Big Data in Your Small Business: 5 Tips

Minda Zetlin

Giant corporations use data analysis to fine-tune their marketing. So can you.

You can’t open a newspaper or turn on NPR these days without encountering the word data. Data is the new oil. With data, Target can tell when a woman is pregnant, even before she’s told her own family. Data will drive everything in marketing from now on.

Frequently, though, the word “data” is preceded by the word “big.” Mega-corporations with billions of “data points” can clearly turn data to their advantage, but can you? Can a small company use big data techniques to make its own marketing more effective?

Yes, according to Jesse Pujji, co-founder and president of Ampush Media, which helps brands like Walgreen’s, State Farm, and Milky Way optimize their advertising on Facebook and other online venues. (The other two founders are Chris Amos and Nick Shah; “Ampush” combines the first two letters of its three founders’ names.) You may not have a statistician on hand, but you can still draw value from the data from your online advertising. Here’s how:

1. Measure everything.

Many marketers don’t take their analysis far enough, Pujji says. They’ll measure how many people clicked on a link, but not what they were doing before they did. Or they’ll measure how long potential customers stay on a site, but not how long they looked at each page.

In one case, he says, Ampush analyzed a mortgage site. Visitors had to input information in several steps in order to be connected to a mortgage counselor. “We started to look at the time between steps,” Pujji says. “We noticed that for more than 5% of visitors, if they hesitated more than two to three milliseconds at a step, we would lose them. We set up the site so that whenever there was that kind of hesitation, it would deliver an inspirational message, such as “Low rates are just a few clicks away.” That improved conversion rates by 30 to 40%.”

2. Make sure to consider outcome.

In online advertising, much of the attention goes to when and whether a user clicks on an ad. For one thing, that’s what triggers payment in a pay-per-click system. But you don’t truly benefit, Pujji points out, until someone actually buys something. “There’s click, then action–such as signing up for a newsletter or requesting more information–and outcome, when a transaction actually takes place,” Pujji says. It’s important to measure all three.

3. Focus on good customers.

Even beyond outcome, ask whether the customer you acquired with your ad is a good one or a bad one. “A lot of platforms tie together different parts of data, but they don’t differentiate among customers,” he says. “People buying keywords on Google might say, ‘I had this many clicks, and this many conversions.’ But did they go on to become good customers? And is there a keyword that drove high-value customers to your site, versus low-value ones?”

4. Start small.

Although measuring everything is sound advice, don’t expect to do it on the first day, Pujji says. “It’s OK to start slow and get it right at a small level rather than a big one. Get successful with a handful of ads, perhaps five, before you start to do any kind of scaling. People look at these platforms and either get super intimidated or else they try to go too far too soon.”

5. Be an early adopter.

One natural advantage for a small company is that it can try out new markets and new social media more quickly than a slow-moving mega-corporation. So consider making use of that advantage whenever new marketing opportunities come along.

“I would advise people to explore the new platforms as they emerge,” Pujji says. “Facebook has been really interesting because of the ability to really target age, interests and demographics. Twitter will launch an advertising platform soon; Tumblr is just starting to do it.” [Tumblr founder David Karp announced Tumblr would start featuring sponsored posts on May 2.] “A year from now, Pinterest will have launched something. Try to be on the bleeding edge, if you can.”


Recommended for you: