Sections

Cathy O’Neil: Unmasking Unconscious Bias in Algorithms

Fawn Fitter

In the wake of the 2008 banking crisis, Cathy O’Neil, a former Barnard College math professor turned hedge fund data scientist, realized that the algorithms she once believed would solve complex problems with pure logic were instead creating them at great speed and scale. Now O’Neil—who goes by mathbabe on her popular blog and 11,000-follower Twitter account—works at bringing to light the dark side of Big Data: mathematical models that operate without transparency, without regulation, and—worst of all—without recourse if they’re wrong. She’s the founder of the Lede Program for Data Journalism at Columbia University, and her bestselling book, Weapons of Math Destruction (Crown, 2016), was long-listed for the 2016 National Book Award.

We asked O’Neil about creating accountability for mathematical models that businesses use to make critical decisions.

Q. If an algorithm applies rules equally across the board, how can the results be biased?

Cathy O’Neil: Algorithms aren’t inherently fair or trustworthy just because they’re mathematical. “Garbage in, garbage out” still holds.

There are many examples: On Wall Street, the mortgage-backed security algorithms failed because they were simply a lie. A program designed to assess teacher performance based only on test results fails because it’s just bad statistics; moreover, there’s much more to learning than testing. A tailored advertising startup I worked for created a system that served ads for things users wanted, but for-profit colleges used that same infrastructure to identify and prey on low-income single mothers who could ill afford useless degrees. Models in the justice system that recommend sentences and predict recidivism tend to be based on terribly biased policing data, particularly arrest records, so their predictions are often racially skewed.

Q. Does bias have to be introduced deliberately for an algorithm to make skewed predictions?

O’Neil: No! Imagine that a company with a history of discriminating against women wants to get more women into the management pipeline and chooses to use a machine-learning algorithm to select potential hires more objectively. They train that algorithm with historical data about successful hires from the last 20 years, and they define successful hires as people they retained for 5 years and promoted at least twice.

They have great intentions. They aren’t trying to be biased; they’re trying to mitigate bias. But if they’re training the algorithm with past data from a time when they treated their female hires in ways that made it impossible for them to meet that specific definition of success, the algorithm will learn to filter women out of the current application pool, which is exactly what they didn’t want.

I’m not criticizing the concept of Big Data. I’m simply cautioning everyone to beware of oversized claims about and blind trust in mathematical models.

Q. What safety nets can business leaders set up to counter bias that might be harmful to their business?

O’Neil: They need to ask questions about, and support processes for, evaluating the algorithms they plan to deploy. As a start, they should demand evidence that an algorithm works as they want it to, and if that evidence isn’t available, they shouldn’t deploy it. Otherwise they’re just automating their problems.

Once an algorithm is in place, organizations need to test whether their data models look fair in real life. For example, the company I mentioned earlier that wants to hire more women into its management pipeline could look at the proportion of women applying for a job before and after deploying the algorithm. If applications drop from 50% women to 25% women, that simple measurement is a sign something might be wrong and requires further checking.

Very few organizations build in processes to assess and improve their algorithms. One that does is Amazon: Every single step of its checkout experience is optimized, and if it suggests a product that I and people like me don’t like, the algorithm notices and stops showing it. It’s a productive feedback loop because Amazon pays attention to whether customers are actually taking the algorithm’s suggestions.

Q. You repeatedly warn about the dangers of using machine learning to codify past mistakes, essentially, “If you do what you’ve always done, you’ll get what you’ve always gotten.” What is the greatest risk companies take when trusting their decision making to data models?

O’Neil: The greatest risk is to trust the data model itself not to expose you to risk, particularly legally actionable risk. Any time you’re considering using an algorithm under regulated conditions, like hiring, promotion, or surveillance, you absolutely must audit it for legality. This seems completely obvious; if it’s illegal to discriminate against people based on certain criteria, for example, you shouldn’t use an algorithm that does so! And yet companies often use discriminatory algorithms because it doesn’t occur to them to ask about it, or they don’t know the right questions to ask, or the vendor or developer hasn’t provided enough visibility into the algorithm for the question to be easily answered.

Q. What are the ramifications for businesses if they persist in believing that data is neutral?

O’Neil: As more evidence comes out that poorly designed algorithms cause problems, I think that people who use them are going to be held accountable for bad outcomes. The era of plausible deniability for the results of using Big Data—that ability to say they were generated without your knowledge—is coming to an end. Right now, algorithm-based decision making is a few miles ahead of lawyers and regulations, but I don’t think that’s going to last. Regulators are already taking steps toward auditing algorithms for illegal properties.

Whenever you use an automated system, it generates a history of its use. If you use an algorithm that’s illegally biased, the evidence will be there in the form of an audit trail. This is a permanent record, and we need to think about our responsibility to ensure it’s working well. D!

Comments

Supply Chain Risk Managers Must Be Fuzzy, Or Fail

Susan Galer

I’ve been eager for the chance to inject the fashionable word “fraught” into one of my blogs for a while, and a recent conversation about supply chain risk presented the ideal opportunity.

During an exclusive roundtable at the SAP Ariba Live 2017 event in Las Vegas entitled “Managing Risk in Your Supplier Engagements,” three experts talked about how companies can prevent the worst from happening in a world fraught with stuff that can go wrong.

Their message was that companies can use advanced technologies like machine learning and predictive analytics to neutralize the impact of natural disasters, global currency fluctuations, and labor strikes, more easily ensure compliance with increasing regulations, and even address evils like forced and slave labor in their supply chain ─ but only if all that tech is backed by a corporate commitment to do good.

Cognitive computing changes the game for risk managers

Investigators and risk managers require both data transparency and context, something Padmini Ranganathan, vice president, products & innovation at SAP Ariba, said is foundational to how the SAP Ariba network of buyers and sellers operates. Dan Adamson, CEO of OutsideIQ, an SAP Ariba partner, discussed his company’s cognitive computing platform, which, together with SAP Ariba, changes the game.

Ranganathan noted that advanced technologies can help companies make sure they have the right data at the right time in the right place, and with the right person able to act. “When your supplier is tripping up somewhere, you need to be there to catch it,” he advised. “Technology is a very powerful tool with the ability to machine learn and pattern match to find out what’s going on.”

“Until now, machines have been great at combing through vast amounts of data but not providing context,” he added. “We bring in the right data and apply the first layer of context to make sure it’s a risk you would care about. How you deal with it is another level of context. We’ll see an evolution because some of your suppliers, depending on your industry, might have a heavy regulatory slant, and you need to treat them differently. Our layers of cognitive computing help filter out the noise and bring the relevant events to bear.”

Outside IQ conducts research far beyond simple watch list monitoring. “We go deeper with our cognitive process, replicating what a researcher would do, looking for patterns and links,” Ranganathan continued. “What might be clean today may have a news report tomorrow. Companies need to know before something becomes an explosive issue. The power SAP Ariba brings in is the whole layer of scoring indicators with relationship insights.”

Purpose-driven supply chain

James Edward Johnson, director of supply risk and analytics at Nielsen, said companies have a shared responsibility in managing supply chains for the greater good. The SAP Ariba network helps Nielsen conduct due diligence at scale faster and more cost-efficiently.

“World development has made some people richer and left a lot of people behind,” Johnson noted. “Because we’re so active in the supply chain, we actually touch millions of lives. How do you make sure that’s a force for good, that when you negotiate deals your push for price isn’t merely favoring companies that will cut corners, abuse their workers, enslave people, or rip up the environment by dumping chemicals into lakes?

“SAP Ariba is a great platform because it’s to a degree, data-neutral. A group like Outside IQ will find and read documents from everywhere in the world. If we can find and solve problems in our supply chain, we can make a difference in the world.”

Forget focus, follow the arc to uncover bad behavior

Responding to an audience member question, Johnson cautioned against zeroing in on risks.

“The moment you start focusing, you’re going to fail to capture risk, which is about seeing the unseen,” he said. “Sometimes your peripheral vision is more effective than your central vision. This is the arc of whatever risk you’re looking at. For example, I can guarantee financial indicators are a good leading indicator. The moment a company starts to fail at meeting their numbers, they’ll start taking risks. The question is where those risks materialize. You have look at other things that might provoke bad behavior.”

Every risk manager should be willing to say, “The answer I just gave you is wrong.”

Make data actionable, but accept fuzziness

These experts agreed that people need to factor risk indicators into contract negotiations while recognizing the level of uncertainty inherent to all kinds of data.

“Everyone in risk management should be willing to say ‘the answer I just gave you is wrong’ – the question is by how much and in what direction,” said Johnson. “Too often people are called on to give specific answers they can hang their hat on. That might teach people to manipulate the data or give people who are politically capable an advantage over people who are technically capable, so you might end up promoting people who are better at talking.”

Machine learning promises to strip out biases like recency and sample selection to give decision makers greater objectivity in understanding actual and potential risks and how to address them. “We should have science-based answers, we should have the data, and we should be able to know how well we know what we say we know,” said Johnson.

For more supply chain risk management strategies, see Managing Third-Party Risk Through Verified Trust.

Follow me: @smgaler

Comments

How The World Is Changing, And How IoT Can Change The World

Richard Howells

In 2016, the world population surpassed 7.3 billion, meaning the number of people on Earth has doubled in the past 45 years.

Furthermore, there’s no end insight to this rapid growth. In fact, it has been calculated that the population is increasing by more than 140 people every minute.

This exploding population creates challenges from the perspectives of sustainability and urbanization:

  • The world’s population is outgrowing the available natural resources. If, as predicted, the global population reaches 9.6 billion by 2050, we would, per United Nations figures, require the equivalent of almost three planets to provide the natural resources needed to sustain current lifestyles.
  • Global population is migrating to urban areas, creating a strain on infrastructure. The United Nations predicts that by 2030, 60% of the world population will live in urban areas. In 2000, this number was only 47%. Nearly 180,000 people are added to the urban population each day.

These two facts alone suggest we must innovate to leverage our natural resources in a more sustainable way, and in many cases, do more with less.

IoT revolutionizes farming

In a recent blog post, SAP and Stara: IoT’s New Constellation of Stars, author Judith Magyar explains how tractors can now leverage sensors to capture real-time data and optimize farm management and operations.

By leveraging data analytic capabilities, farmers can:

  • Gain better insights to make more sound decisions
  • Produce more high-quality crops per acre
  • Support sustainable agricultural practices that respect the environment

This is just one example of how the Internet of Things (IoT) can help balance supply and demand for the world’s natural resources to help maximize production, reduce environmental impact, and remediate problems faster.

Check out this video to learn more about how IoT is revolutionizing the farming industry.

Argentina modernizes its capital city with IoT

In another recent article, 3 Ways Buenos Aires Is Leading Smart City Technology, author Christine Donato explains how the Argentinian capital is leveraging IoT solutions to mitigate some of the challenges of a growing urban area.

This includes:

  • Managing and responding to the 30,000 complaints the city receives each month by leveraging real-time data and social media insight
  • Reducing consumption and keeping the city illuminated and safe by converting 91,000 public street lights to modern LED technology, and using real-time insights to quickly address power outages, broken lights, and vandalism
  • Automating maintenance on 1,400 kilometers of drainage pipes to execute cleaning and maintenance at maximum capacity and predict potential flooding

Check out this video to learn more about how Buenos Aires is illuminating the city in a more sustainable way.

These are just two examples of how organizations are leveraging IoT to change the world. For more on these stories and many others, attend the SAP Leonardo Live – IoT for Business event on July 11 and 12 in Frankfurt, Germany. Visit this website for additional details.

Comments

About Richard Howells

Richard Howells is a Vice President at SAP responsible for the positioning, messaging, AR , PR and go-to market activities for the SAP Supply Chain solutions.

The Future of Cybersecurity: Trust as Competitive Advantage

Justin Somaini and Dan Wellers

 

The cost of data breaches will reach US$2.1 trillion globally by 2019—nearly four times the cost in 2015.

Cyberattacks could cost up to $90 trillion in net global economic benefits by 2030 if cybersecurity doesn’t keep pace with growing threat levels.

Cyber insurance premiums could increase tenfold to $20 billion annually by 2025.

Cyberattacks are one of the top 10 global risks of highest concern for the next decade.


Companies are collaborating with a wider network of partners, embracing distributed systems, and meeting new demands for 24/7 operations.

But the bad guys are sharing intelligence, harnessing emerging technologies, and working round the clock as well—and companies are giving them plenty of weaknesses to exploit.

  • 33% of companies today are prepared to prevent a worst-case attack.
  • 25% treat cyber risk as a significant corporate risk.
  • 80% fail to assess their customers and suppliers for cyber risk.

The ROI of Zero Trust

Perimeter security will not be enough. As interconnectivity increases so will the adoption of zero-trust networks, which place controls around data assets and increases visibility into how they are used across the digital ecosystem.


A Layered Approach

Companies that embrace trust as a competitive advantage will build robust security on three core tenets:

  • Prevention: Evolving defensive strategies from security policies and educational approaches to access controls
  • Detection: Deploying effective systems for the timely detection and notification of intrusions
  • Reaction: Implementing incident response plans similar to those for other disaster recovery scenarios

They’ll build security into their digital ecosystems at three levels:

  1. Secure products. Security in all applications to protect data and transactions
  2. Secure operations. Hardened systems, patch management, security monitoring, end-to-end incident handling, and a comprehensive cloud-operations security framework
  3. Secure companies. A security-aware workforce, end-to-end physical security, and a thorough business continuity framework

Against Digital Armageddon

Experts warn that the worst-case scenario is a state of perpetual cybercrime and cyber warfare, vulnerable critical infrastructure, and trillions of dollars in losses. A collaborative approach will be critical to combatting this persistent global threat with implications not just for corporate and personal data but also strategy, supply chains, products, and physical operations.


Download the executive brief The Future of Cybersecurity: Trust as Competitive Advantage.


Comments

Tags:

How Digital Transformation Is Rewriting Business Models

Ginger Shimp

Everybody knows someone who has a stack of 3½-inch floppies in a desk drawer “just in case we may need them someday.” While that might be amusing, the truth is that relatively few people are confident that they’re making satisfactory progress on their digital journey. The boundaries between the digital and physical worlds continue to blur — with profound implications for the way we do business. Virtually every industry and every enterprise feels the effects of this ongoing digital transformation, whether from its own initiative or due to pressure from competitors.

What is digital transformation? It’s the wholesale reimagining and reinvention of how businesses operate, enabled by today’s advanced technology. Businesses have always changed with the times, but the confluence of technologies such as mobile, cloud, social, and Big Data analytics has accelerated the pace at which today’s businesses are evolving — and the degree to which they transform the way they innovate, operate, and serve customers.

The process of digital transformation began decades ago. Think back to how word processing fundamentally changed the way we write, or how email transformed the way we communicate. However, the scale of transformation currently underway is drastically more significant, with dramatically higher stakes. For some businesses, digital transformation is a disruptive force that leaves them playing catch-up. For others, it opens to door to unparalleled opportunities.

Upending traditional business models

To understand how the businesses that embrace digital transformation can ultimately benefit, it helps to look at the changes in business models currently in process.

Some of the more prominent examples include:

  • A focus on outcome-based models — Open the door to business value to customers as determined by the outcome or impact on the customer’s business.
  • Expansion into new industries and markets — Extend the business’ reach virtually anywhere — beyond strictly defined customer demographics, physical locations, and traditional market segments.
  • Pervasive digitization of products and services — Accelerate the way products and services are conceived, designed, and delivered with no barriers between customers and the businesses that serve them.
  • Ecosystem competition — Create a more compelling value proposition in new markets through connections with other companies to enhance the value available to the customer.
  • Access a shared economy — Realize more value from underutilized sources by extending access to other business entities and customers — with the ability to access the resources of others.
  • Realize value from digital platforms — Monetize the inherent, previously untapped value of customer relationships to improve customer experiences, collaborate more effectively with partners, and drive ongoing innovation in products and services,

In other words, the time-tested assumptions about how to identify customers, develop and market products and services, and manage organizations may no longer apply. Every aspect of business operations — from forecasting demand to sourcing materials to recruiting and training staff to balancing the books — is subject to this wave of reinvention.

The question is not if, but when

These new models aren’t predictions of what could happen. They’re already realities for innovative, fast-moving companies across the globe. In this environment, playing the role of late adopter can put a business at a serious disadvantage. Ready or not, digital transformation is coming — and it’s coming fast.

Is your company ready for this sea of change in business models? At SAP, we’ve helped thousands of organizations embrace digital transformation — and turn the threat of disruption into new opportunities for innovation and growth. We’d relish the opportunity to do the same for you. Our Digital Readiness Assessment can help you see where you are in the journey and map out the next steps you’ll need to take.

Up next I’ll discuss the impact of digital transformation on processes and work. Until then, you can read more on how digital transformation is impacting your industry.

Comments

About Ginger Shimp

With more than 20 years’ experience in marketing, Ginger Shimp has been with SAP since 2004. She has won numerous awards and honors at SAP, including being designated “Top Talent” for two consecutive years. Not only is she a Professional Certified Marketer with the American Marketing Association, but she's also earned her Connoisseur's Certificate in California Reds from the Chicago Wine School. She holds a bachelor's degree in journalism from the University of San Francisco, and an MBA in marketing and managerial economics from the Kellogg Graduate School of Management at Northwestern University. Personally, Ginger is the proud mother of a precocious son and happy wife of one of YouTube's 10 EDU Gurus, Ed Shimp.