Sections

Cathy O’Neil: Unmasking Unconscious Bias in Algorithms

Fawn Fitter

In the wake of the 2008 banking crisis, Cathy O’Neil, a former Barnard College math professor turned hedge fund data scientist, realized that the algorithms she once believed would solve complex problems with pure logic were instead creating them at great speed and scale. Now O’Neil—who goes by mathbabe on her popular blog and 11,000-follower Twitter account—works at bringing to light the dark side of Big Data: mathematical models that operate without transparency, without regulation, and—worst of all—without recourse if they’re wrong. She’s the founder of the Lede Program for Data Journalism at Columbia University, and her bestselling book, Weapons of Math Destruction (Crown, 2016), was long-listed for the 2016 National Book Award.

We asked O’Neil about creating accountability for mathematical models that businesses use to make critical decisions.

Q. If an algorithm applies rules equally across the board, how can the results be biased?

Cathy O’Neil: Algorithms aren’t inherently fair or trustworthy just because they’re mathematical. “Garbage in, garbage out” still holds.

There are many examples: On Wall Street, the mortgage-backed security algorithms failed because they were simply a lie. A program designed to assess teacher performance based only on test results fails because it’s just bad statistics; moreover, there’s much more to learning than testing. A tailored advertising startup I worked for created a system that served ads for things users wanted, but for-profit colleges used that same infrastructure to identify and prey on low-income single mothers who could ill afford useless degrees. Models in the justice system that recommend sentences and predict recidivism tend to be based on terribly biased policing data, particularly arrest records, so their predictions are often racially skewed.

Q. Does bias have to be introduced deliberately for an algorithm to make skewed predictions?

O’Neil: No! Imagine that a company with a history of discriminating against women wants to get more women into the management pipeline and chooses to use a machine-learning algorithm to select potential hires more objectively. They train that algorithm with historical data about successful hires from the last 20 years, and they define successful hires as people they retained for 5 years and promoted at least twice.

They have great intentions. They aren’t trying to be biased; they’re trying to mitigate bias. But if they’re training the algorithm with past data from a time when they treated their female hires in ways that made it impossible for them to meet that specific definition of success, the algorithm will learn to filter women out of the current application pool, which is exactly what they didn’t want.

I’m not criticizing the concept of Big Data. I’m simply cautioning everyone to beware of oversized claims about and blind trust in mathematical models.

Q. What safety nets can business leaders set up to counter bias that might be harmful to their business?

O’Neil: They need to ask questions about, and support processes for, evaluating the algorithms they plan to deploy. As a start, they should demand evidence that an algorithm works as they want it to, and if that evidence isn’t available, they shouldn’t deploy it. Otherwise they’re just automating their problems.

Once an algorithm is in place, organizations need to test whether their data models look fair in real life. For example, the company I mentioned earlier that wants to hire more women into its management pipeline could look at the proportion of women applying for a job before and after deploying the algorithm. If applications drop from 50% women to 25% women, that simple measurement is a sign something might be wrong and requires further checking.

Very few organizations build in processes to assess and improve their algorithms. One that does is Amazon: Every single step of its checkout experience is optimized, and if it suggests a product that I and people like me don’t like, the algorithm notices and stops showing it. It’s a productive feedback loop because Amazon pays attention to whether customers are actually taking the algorithm’s suggestions.

Q. You repeatedly warn about the dangers of using machine learning to codify past mistakes, essentially, “If you do what you’ve always done, you’ll get what you’ve always gotten.” What is the greatest risk companies take when trusting their decision making to data models?

O’Neil: The greatest risk is to trust the data model itself not to expose you to risk, particularly legally actionable risk. Any time you’re considering using an algorithm under regulated conditions, like hiring, promotion, or surveillance, you absolutely must audit it for legality. This seems completely obvious; if it’s illegal to discriminate against people based on certain criteria, for example, you shouldn’t use an algorithm that does so! And yet companies often use discriminatory algorithms because it doesn’t occur to them to ask about it, or they don’t know the right questions to ask, or the vendor or developer hasn’t provided enough visibility into the algorithm for the question to be easily answered.

Q. What are the ramifications for businesses if they persist in believing that data is neutral?

O’Neil: As more evidence comes out that poorly designed algorithms cause problems, I think that people who use them are going to be held accountable for bad outcomes. The era of plausible deniability for the results of using Big Data—that ability to say they were generated without your knowledge—is coming to an end. Right now, algorithm-based decision making is a few miles ahead of lawyers and regulations, but I don’t think that’s going to last. Regulators are already taking steps toward auditing algorithms for illegal properties.

Whenever you use an automated system, it generates a history of its use. If you use an algorithm that’s illegally biased, the evidence will be there in the form of an audit trail. This is a permanent record, and we need to think about our responsibility to ensure it’s working well. D!

Comments

Consumer-Driven Digital Enterprise: The Digital Future Of Consumer Products

Jim Cook

Across industries, there’s a lot of talk about how digital is rewriting the rules of engagement.

We are shown examples of how digital disruption is impacting almost every aspect of businesses – from reinventing business models to transforming business processes. Re-imagining a business platform is almost a requirement in today’s consumer-led and data-driven economy. (You know the businesses that are quoted in every article on digital.)

A key question here, though, is: whether the consumer products industry is indeed facing digital disruption, or does it really need deeper digital innovation? Disruption turns an industry on its head by offering consumers something that previously did not exist, while innovation enhances an existing value proposition – making it better, faster, or cheaper.

It is important to distinguish between the two, because hype often causes businesses to overlook the true value of digital transformation. Companies may presume such radical changes have nothing to do with them, especially if they are already in a dominant market position. So while digital is dramatically changing industries such as retail and healthcare, the disruption in the consumer product industry may not be as severe – not yet anyway. Instead, what consumer products businesses should focus on is how they can transform digitally to gain the capacity to build and grow “live brands.” This is preparation and not protectionism.

Create direct customer experiences: Secure the dominant market position

The digital age has fundamentally shifted customer and consumer expectations. Consumers increasingly value outcomes over products. To build ongoing engagement and loyalty, consumer products companies need to sense and engage consumers and customers in the moment, i.e. build “live brands” by seamlessly delivering highly personalized experiences – anytime, anywhere.

This ability to create direct customer experiences helps consumer products companies create a sharper competitive edge to secure dominant market positions. Leading consumer products companies know this well.

Red Bull sets a fine example in creating direct customer experiences to protect and strengthen its brand. Today, it has moved beyond a beverage company into a content media company spanning web, social, film, print, music, and TV – creating brand experiences of exhilaration and adventure. Red Bull collects data from every touch point that it has with the consumer, building an enhanced profile of every individual so that it can respond with products that consumers desire – whenever and how they want them.

Procter & Gamble recently launched an online, direct-to-consumer subscription business for its Tide Pods (its highest-priced laundry detergent). The service (currently only available in Atlanta), branded Tide On Demand, offers free shipping of Tide Pods at regular intervals. P&G has also been testing its delivery laundry service – Tide Spin – in Chicago. While the direct-to-consumer services may not form a bulk of its revenue, they allow P&G to quickly build a live understanding of its customers, their preferences and habits, and then hone in on these insights to create new offerings that customers want.

Build a real-time supply chain: Support lasting customer loyalty

As consumer products companies move towards sensing and engaging customers in the moment, they also need to ensure a fast and profitable response to dynamic demand.

This necessitates connecting customer insights that brand owners have collated and analyzed with supply chain insights to accelerate time to market. Ultimately, it is about transforming previously linear supply chains into customer-centric demand networks – where demand information is captured through new signals from various sources (such as retailers, wholesalers, sites like Amazon, directly from customers, or the Internet of Things) and fulfilled through the orchestration of a network of internal and external partners.

With that, consumer product companies can start getting answers to questions such as:

  • What are my short-, mid-, and long-term views of expected demand across channels?
  • How can I combine supply chain planning with strategic, financial, sales, and operational goals?
  • How can I extend planning by collaborating with customers, partners, and suppliers?
  • How can the company translate the plan into actionable targets for fulfillment systems?

All these should go full circle to help make manufacturing more responsive, optimizing capacity to help ensure availability of finished goods produced just-in-time to meet demand, thereby also lowering inventory costs.

Consumer products companies need to consider how they can create the digital future today. We invite you to learn more about digital transformation for the consumer products industry, where you will get access to valuable resources including whitepapers and customer case studies.

Comments

Jim Cook

About Jim Cook

Jim Cook is the Industry Advisor for consumer industries in South East Asia, with over 20 years’ experience of IT and business consulting. He has held various roles from solution architect, project and program management, business development as well as managing an SAP partner organisation. Jim is passionate about transformation within consumer driven organisations. Jim is particular interested in customer engagement solutions and the value that can be achieved from end to end SAP deployments.

Live Product Innovation, Part 2: IoT, Big Data, and Smart Connected Products

John McNiff

In Part 1 of this series, we looked at how in-memory computing affects live product innovation. In Part 2, we explore the impact of the Internet of Things (IoT) and Big Data on smart connected products. In Part 3, we’ll approach the topic from the perspective of process industries.

Live engineering? Live product innovation? Live R&D?

To some people, these concepts sound implausible. When you talk about individualized product launches with lifecycles of days or weeks, people in industries like aerospace and defense (A&D) look askance.

But today, most industries—not just consumer-driven ones—need timely insights and the ability to respond quickly. Even A&D manufacturers want to understand the impact of changes before they continue with designs that could be difficult to make at the right quantity or prone to problems in the field.

The Internet of Everything?

Internet of Things (IoT) technologies promise to give manufacturers these insights. But there’s still a lot of confusion around IoT. Some people think it is about connected appliances; others think it’s just a rebranded “shop floor to top floor.”

The better way to think about IoT is from the perspective of data: We want to get data from the connected “thing.” If you’re the manufacturer, that thing is a product. If you buy the product and sell it to an end customer, that thing is an asset. If you’re the end customer, that thing is a fleet. Each stakeholder wants different data in different volumes for different reasons.

It’s also important to remember that the Internet has existed for far longer than IoT.

There’s a huge amount of non-IoT data that can offer useful insights. Point-of-sale data, news feeds, and market insights from social channels are all valuable. And think about how much infrastructure is now connected in “smart” cities. So in addition to products, assets, and fleets, there are also people, markets, and infrastructure. Big Data is everywhere, and it should influence what you release and when.

New data, new processes

It has been said that data in the 21st century is like oil in the 18th century: an immense, valuable, yet untapped asset. But if data is the new oil, then do we need a new refinery? The answer is yes.

On top of business data, we now have a plethora of information sources outside our company walls. Ownership of, and access to, this data is becoming complex. Manufacturers collecting data about equipment at customer sites, for example, may want to sell that data to customers as an add-on service. But those customers are likely using equipment from multiple manufacturers, and they likely have their own unique uses for the information.

So the new information refinery needs to capture information from everywhere and turn it into something that has meaning for the end user. It needs to leverage data science and machine learning to remove the noise and add insight and intelligence. It also needs to be an open platform to gather information from all six sources (products, assets, fleets, people, markets, infrastructure).

And wouldn’t it be great if the data refinery ran on the same platform as your business processes, so that you could sense, respond, and act to achieve your business goals?

Digital product innovation platform

If you start with the concept of a smart connected product, the data refinery — the digital product innovation platform — has five requirements:

  1. Systems design — Manufacturers need to design across disciplines in a systems approach. Mechanical, mechatronic, electrical, electronics, and software all need to be supported, with modeling capabilities that cover physical, functional, and logical structures.
  1. Requirements-linked platform design — Designers need to think about where and how to embed sensors and intelligence to match functional requirements. This will need to be forward-thinking to cover unforeseen methods of machine-to-machine interactions. In a world of performance-based contracts, it will be important to minimize the impact of design changes as innovation opportunities grow.
  1. Instant impact and insight environment — The platform must support fully traceable requirements throughout the lifecycle, from design concept to asset performance.
  1. Product-based enterprise processes — The platform needs to share model-based product data visually — through electrical CAD, electronic CAD, 3D, and software functions — to the people who need it. This isn’t new, but what’s different is that the platform can’t wait for complex integrations between systems. Think about software-enabled innovation or virtual inventory made possible by on-demand 3D printing. Production is almost real time, so design will have to be as well.
  1. Product and thing network — A complex, cross-domain design process involves a growing number of partners. That calls for a product network to allow for secure collaboration across functions and outside the company walls. Instead of every partner having its own portal for product data, the product network would store digital twins and allow instant sharing of asset intelligence.

If the network is connected to the digital product innovation platform, you can control the lifecycle both internally and externally — and take the product right into the service and maintenance domain. You can then provide field information directly from the assets back to design to inform what to update, and when. Add over-the-air software compatibility checks and updates, and discrete manufacturers can achieve a true live engineering environment.

Sound like a dream? It’s coming sooner than you think.

Learn more about supply chain innovations at SAP.com or follow us on @SCMatSAP for the latest news.

Comments

John McNiff

About John McNiff

John McNiff is the Vice President of Solution Management for the R&D/Engineering line-of-business business unit at SAP. John has held a number of sales and business development roles at SAP, focused on the manufacturing and engineering topics.

How Emotionally Aware Computing Can Bring Happiness to Your Organization

Christopher Koch


Do you feel me?

Just as once-novel voice recognition technology is now a ubiquitous part of human–machine relationships, so too could mood recognition technology (aka “affective computing”) soon pervade digital interactions.

Through the application of machine learning, Big Data inputs, image recognition, sensors, and in some cases robotics, artificially intelligent systems hunt for affective clues: widened eyes, quickened speech, and crossed arms, as well as heart rate or skin changes.




Emotions are big business

The global affective computing market is estimated to grow from just over US$9.3 billion a year in 2015 to more than $42.5 billion by 2020.

Source: “Affective Computing Market 2015 – Technology, Software, Hardware, Vertical, & Regional Forecasts to 2020 for the $42 Billion Industry” (Research and Markets, 2015)

Customer experience is the sweet spot

Forrester found that emotion was the number-one factor in determining customer loyalty in 17 out of the 18 industries it surveyed – far more important than the ease or effectiveness of customers’ interactions with a company.


Source: “You Can’t Afford to Overlook Your Customers’ Emotional Experience” (Forrester, 2015)


Humana gets an emotional clue

Source: “Artificial Intelligence Helps Humana Avoid Call Center Meltdowns” (The Wall Street Journal, October 27, 2016)

Insurer Humana uses artificial intelligence software that can detect conversational cues to guide call-center workers through difficult customer calls. The system recognizes that a steady rise in the pitch of a customer’s voice or instances of agent and customer talking over one another are causes for concern.

The system has led to hard results: Humana says it has seen an 28% improvement in customer satisfaction, a 63% improvement in agent engagement, and a 6% improvement in first-contact resolution.


Spread happiness across the organization

Source: “Happiness and Productivity” (University of Warwick, February 10, 2014)

Employers could monitor employee moods to make organizational adjustments that increase productivity, effectiveness, and satisfaction. Happy employees are around 12% more productive.




Walking on emotional eggshells

Whether customers and employees will be comfortable having their emotions logged and broadcast by companies is an open question. Customers may find some uses of affective computing creepy or, worse, predatory. Be sure to get their permission.


Other limiting factors

The availability of the data required to infer a person’s emotional state is still limited. Further, it can be difficult to capture all the physical cues that may be relevant to an interaction, such as facial expression, tone of voice, or posture.



Get a head start


Discover the data

Companies should determine what inferences about mental states they want the system to make and how accurately those inferences can be made using the inputs available.


Work with IT

Involve IT and engineering groups to figure out the challenges of integrating with existing systems for collecting, assimilating, and analyzing large volumes of emotional data.


Consider the complexity

Some emotions may be more difficult to discern or respond to. Context is also key. An emotionally aware machine would need to respond differently to frustration in a user in an educational setting than to frustration in a user in a vehicle.

 


 

download arrowTo learn more about how affective computing can help your organization, read the feature story Empathy: The Killer App for Artificial Intelligence.


Comments

About Christopher Koch

Christopher Koch is the Editorial Director of the SAP Center for Business Insight. He is an experienced publishing professional, researcher, editor, and writer in business, technology, and B2B marketing. Share your thoughts with Chris on Twitter @Ckochster.

Tags:

In An Agile Environment, Revenue Models Are Flexible Too

Todd Wasserman

In 2012, Dollar Shave Club burst on the scene with a cheeky viral video that won praise for its creativity and marketing acumen. Less heralded at the time was the startup’s pricing model, which swapped traditional retail for subscriptions.

For as low as $1 a month (for five two-bladed cartridges), consumers got a package in the mail that saved them a trip to the pharmacy or grocery store. Dollar Shave Club received the ultimate vindication for the idea in 2016 when Unilever purchased the company for $1 billion.

As that example shows, new technology creates the possibility for new pricing models that can disrupt existing industries. The same phenomenon has occurred in software, in which the cloud and Web-based interfaces have ushered in Software as a Service (SaaS), which charges users on a monthly basis, like a utility, instead of the typical purchase-and-later-upgrade model.

Pricing, in other words, is a variable that can be used to disrupt industries. Other options include usage-based pricing and freemium.

Products as services, services as products

There are basically two ways that businesses can use pricing to disrupt the status quo: Turn products into services and turn services into products. Dollar Shave Club and SaaS are two examples of turning products into services.

Others include Amazon’s Dash, a bare-bones Internet of Things device that lets consumers reorder items ranging from Campbell’s Soup to Play-Doh. Another example is Rent the Runway, which rents high-end fashion items for a weekend rather than selling the items. Trunk Club offers a twist on this by sending items picked out by a stylist to users every month. Users pay for what they want and send back the rest.

The other option is productizing a service. Restaurant franchising is based on this model. While the restaurant offers food service to consumers, for entrepreneurs the franchise offers guidance and brand equity that can be condensed into a product format. For instance, a global HR firm called Littler has productized its offerings with Littler CaseSmart-Charges, which is designed for in-house attorneys and features software, project management tools, and access to flextime attorneys.

As that example shows, technology offers opportunities to try new revenue models. Another example is APIs, which have become a large source of revenue for companies. The monetization of APIs is often viewed as a side business that encompasses a wholly different pricing model that’s often engineered to create huge user bases with volume discounts.

Not a new idea

Though technology has opened up new vistas for businesses seeking alternate pricing models, Rajkumar Venkatesan, a marketing professor at University of Virginia’s Darden School of Business, points out that this isn’t necessarily a new idea. For instance, King Gillette made his fortune in the early part of the 20th Century by realizing that a cheap shaving device would pave the way for a recurring revenue stream via replacement razor blades.

“The new variation was the Keurig,” said Venkatesan, referring to the coffee machine that relies on replaceable cartridges. “It has started becoming more prevalent in the last 10 years, but the fundamental model has been there.” For businesses, this can be an attractive model not only for the recurring revenue but also for the ability to cross-sell new goods to existing customers, Venkatesan said.

Another benefit to a subscription model is that it can also supply first-party data that companies can use to better understand and market to their customers. Some believe that Dollar Shave Club’s close relationship with its young male user base was one reason for Unilever’s purchase, for instance. In such a cut-throat market, such relationships can fetch a high price.

To learn more about how you can monetize disruption, watch this video overview of the new SAP Hybris Revenue Cloud.

Comments