Cathy O’Neil: Unmasking Unconscious Bias in Algorithms

Fawn Fitter

In the wake of the 2008 banking crisis, Cathy O’Neil, a former Barnard College math professor turned hedge fund data scientist, realized that the algorithms she once believed would solve complex problems with pure logic were instead creating them at great speed and scale. Now O’Neil—who goes by mathbabe on her popular blog and 11,000-follower Twitter account—works at bringing to light the dark side of Big Data: mathematical models that operate without transparency, without regulation, and—worst of all—without recourse if they’re wrong. She’s the founder of the Lede Program for Data Journalism at Columbia University, and her bestselling book, Weapons of Math Destruction (Crown, 2016), was long-listed for the 2016 National Book Award.

We asked O’Neil about creating accountability for mathematical models that businesses use to make critical decisions.

Q. If an algorithm applies rules equally across the board, how can the results be biased?

Cathy O’Neil: Algorithms aren’t inherently fair or trustworthy just because they’re mathematical. “Garbage in, garbage out” still holds.

There are many examples: On Wall Street, the mortgage-backed security algorithms failed because they were simply a lie. A program designed to assess teacher performance based only on test results fails because it’s just bad statistics; moreover, there’s much more to learning than testing. A tailored advertising startup I worked for created a system that served ads for things users wanted, but for-profit colleges used that same infrastructure to identify and prey on low-income single mothers who could ill afford useless degrees. Models in the justice system that recommend sentences and predict recidivism tend to be based on terribly biased policing data, particularly arrest records, so their predictions are often racially skewed.

Q. Does bias have to be introduced deliberately for an algorithm to make skewed predictions?

O’Neil: No! Imagine that a company with a history of discriminating against women wants to get more women into the management pipeline and chooses to use a machine-learning algorithm to select potential hires more objectively. They train that algorithm with historical data about successful hires from the last 20 years, and they define successful hires as people they retained for 5 years and promoted at least twice.

They have great intentions. They aren’t trying to be biased; they’re trying to mitigate bias. But if they’re training the algorithm with past data from a time when they treated their female hires in ways that made it impossible for them to meet that specific definition of success, the algorithm will learn to filter women out of the current application pool, which is exactly what they didn’t want.

I’m not criticizing the concept of Big Data. I’m simply cautioning everyone to beware of oversized claims about and blind trust in mathematical models.

Q. What safety nets can business leaders set up to counter bias that might be harmful to their business?

O’Neil: They need to ask questions about, and support processes for, evaluating the algorithms they plan to deploy. As a start, they should demand evidence that an algorithm works as they want it to, and if that evidence isn’t available, they shouldn’t deploy it. Otherwise they’re just automating their problems.

Once an algorithm is in place, organizations need to test whether their data models look fair in real life. For example, the company I mentioned earlier that wants to hire more women into its management pipeline could look at the proportion of women applying for a job before and after deploying the algorithm. If applications drop from 50% women to 25% women, that simple measurement is a sign something might be wrong and requires further checking.

Very few organizations build in processes to assess and improve their algorithms. One that does is Amazon: Every single step of its checkout experience is optimized, and if it suggests a product that I and people like me don’t like, the algorithm notices and stops showing it. It’s a productive feedback loop because Amazon pays attention to whether customers are actually taking the algorithm’s suggestions.

Q. You repeatedly warn about the dangers of using machine learning to codify past mistakes, essentially, “If you do what you’ve always done, you’ll get what you’ve always gotten.” What is the greatest risk companies take when trusting their decision making to data models?

O’Neil: The greatest risk is to trust the data model itself not to expose you to risk, particularly legally actionable risk. Any time you’re considering using an algorithm under regulated conditions, like hiring, promotion, or surveillance, you absolutely must audit it for legality. This seems completely obvious; if it’s illegal to discriminate against people based on certain criteria, for example, you shouldn’t use an algorithm that does so! And yet companies often use discriminatory algorithms because it doesn’t occur to them to ask about it, or they don’t know the right questions to ask, or the vendor or developer hasn’t provided enough visibility into the algorithm for the question to be easily answered.

Q. What are the ramifications for businesses if they persist in believing that data is neutral?

O’Neil: As more evidence comes out that poorly designed algorithms cause problems, I think that people who use them are going to be held accountable for bad outcomes. The era of plausible deniability for the results of using Big Data—that ability to say they were generated without your knowledge—is coming to an end. Right now, algorithm-based decision making is a few miles ahead of lawyers and regulations, but I don’t think that’s going to last. Regulators are already taking steps toward auditing algorithms for illegal properties.

Whenever you use an automated system, it generates a history of its use. If you use an algorithm that’s illegally biased, the evidence will be there in the form of an audit trail. This is a permanent record, and we need to think about our responsibility to ensure it’s working well. D!

Comments

Fawn Fitter

About Fawn Fitter

Ever since discovering the fledgling Internet in the early 1990s, Fawn Fitter has been fascinated by the places where business and technology intersect. She’s spent 15 years in San Francisco, watching the ebbs and flows of the digital economy and writing for magazines, including Entrepreneur and Fortune Small Business.

How To Speed Up Progress Towards A Digital Supply Chain

Richard Howells

Earlier this year, I wrote about new research from the non-profit organization the Center for Global Enterprise (CGE) describing an important breakthrough in creating a truly digital supply chain. At the time, I focused on how the right people in the right jobs can speed your progress towards a digital supply chain, one that generates demand and revenue while reducing cost.

Now the Digital Supply Chain Institute (the research arm of CGE) has released very practical advice about how to make progress in this important area. Their message is simple: transform your performance measures and you will transform your supply chain. 

Traditional supply chain metrics

Most companies measure supply chain performance by cost and quality. Cost is measured based on “inventory holding cost” and “inventory turns.” Quality is measured based on “perfect orders” and “demand forecast accuracy.” And there are even more measurements for “cycle times” and “capital assets.” These measures drive companies to lower cost and improve delivery performance with low cost manufacturing and logistics.

While this is all very important work, reducing cost and improving quality measures alone will not flip the supply chain management process to focus on the customer. Nor will these actions ensure that the digital supply chain is a source of revenue growth and cost reduction. In fact, George Bailey, the managing director of the Digital Supply Chain Institute, says, “Running an efficient traditional supply chain will lead to loss of market share and revenue decline; and it is game over if a digital native company enters the market!” In order to remain relevant, Bailey insists businesses quickly embrace a new outlook and set of innovative metrics that better assess a company’s transition to digital.

Essential digital supply chain metrics

Many of the traditional supply chain metrics we use today will be just as important in the future. And we all know that we can get better at managing the systems, people, and processes to improve performance against these metrics. However, some of the traditional measures will be dropped to make room for new, essential digital supply chain metrics. Simply adding more measures will not work. The answer is not to create a bigger scorecard with more metrics.  As Anders Karlborg, the assistant CEO for ZTE, stated in the report, “We don’t want to measure how good we are; we want to measure how good our customers think we are.”

The answer to the question of what metrics will shape our progress towards a true customer-facing supply chain is determined by each company’s digital supply chain strategy. It is important for every company to assess where they are, decide where they want to be, determine a time frame and then communicate the vision, strategy and metrics.

Mike Corbo, chief supply chain officer at Colgate-Palmolive, understands the importance of getting this right when he says, “We have a culture that respects and works towards performance metrics. We take them seriously and so do our people.”

Focus on the right measurements

Per the report, there are two types of new measures that will help shape the digital supply chain. The first is an output measure. Output measures reflect a direct impact on a financial statement. These measures capture the results of the work you do on the company’s financial success. For example, a metric that captures incremental revenue from supply chain actions would fall into this category. The second type of measure is a process measure. These record the progress made against process changes. For example, measuring the degree of automation in a firm is a process measure.

The basic rule is that you must have output measures to help you manage the impact of your actions on the company’s financial progress. Also, you must have process measures to keep track of the changes you are making to transform you company and become a truly digital supply chain. Output measures without process measures are empty and process measures without output measures are blind.

Making it happen

Every company wants to create a digital supply chain because the benefits are large. It will take a clear strategy to drive results. It will also take focused investment in technologies that capture real-time Big Data, artificial intelligence, 3D manufacturing, and others. And it will take accountability to deliver results. What is the best way to create accountability? Clear measures, goals, and rewards. The best approach will be for a company to create the digital supply chain strategy, select the metrics, make the investments, and link pay to progress.

As John Waite, VP Global Supply Chain of Micron, states, “As defined by the customer, our business is all about precision, accuracy, and quality, and our metrics also have to be precise, accurate and of high quality.”

Learn more about the CGE research, the new digital supply chain measures that were developed, and the progress that companies are making right here.

Comments

Richard Howells

About Richard Howells

Richard Howells is a Vice President at SAP responsible for the positioning, messaging, AR , PR and go-to market activities for the SAP Supply Chain solutions.

Blockchain: Radically Changing The Mining Paradigm

Indranil Som

The mining industry, with its integrated value chains and conservative views, is going through an intense period of change and can no longer ignore the role of technology in its ecosystem—an ecosystem that is growing and becoming more complex day by day.

Use of technology will change the way miners operate and help companies grow by offering newer business models and delivering improved productivity, safety advancements, and cost savings. These technologies could drive economic transformation in the industry in the coming years and help the mining industry become more competitive globally.

Security is one of the biggest concerns that impacts mining in the future. Blockchain technology can help the mining industry benefit financially and avoid falling prey to security breaches.

What is blockchain technology?

Blockchain, or distributed ledger, is a record of encrypted contracts or transactions that can be accessed using digital keys. It is distributed, replicated, and synchronized across people and locations, and it can be verified by anyone—but it can be changed only when there is a consensus among the group participating in the network.

Because records are stored in the form of a block, and each block has reference to the other block, forming a chain, all the distributed blocks must be hacked simultaneously for an attack to be successful, ensuring high-level of security.

How can blockchain help the mining industry?

There are many areas where blockchain can help the mining companies, including:

1. Improved cybersecurity

According to a report published by security firm Trend Micro, 22 mining companies have reported major cyber-attacks since 2010. These attacks aimed to steal intellectual property and other proprietary information, which can be devastating for any company.

A 2016 Symantec Security report revealed that mining is the number-one industry receiving spam email, and one of three spam emails includes a virus. Blockchain, as a distributed digital ledger, reduces the impact of hacking company-wide by limiting it to only the affected block. Blockchain keeps a record of every transaction and safely encrypts that information without third-party intervention, thereby reducing exposure of data to hackers.

2. Increased transparency with smart contracts

Smart contracts implemented on a blockchain improve transparency between buyers and sellers as goods are tracked in real time from their origin, reducing the chance of fraud, ensuring tracability and transparency, and improving logistics visibility and supply chain quality.

Blockchain can end payment gaps by incorporating delivery and payment in digital contracts and integrating it with logistics partners and banks. Once proof of delivery is received from the logistics team, automatic invoicing and payment can be initiated.

3. Better visibility into supply chain

Blockchain offers more visibility into the supply chain, making procurement and delivery simpler, more accurate, and more reliable. The digital ledger integrates data from all vendors and suppliers across the network, giving a complete picture of the supply chain in real time.

Blockchain brings transparency to the extent that no piece of inventory can exist in the same place at any given point. Transaction status is updated in real time, with full traceability back to the point of origin.

Conclusion

Blockchain technology can radically change transactions by cutting costs to support leaner organizations and increased security. It is a game-changer that offers three major features: security, immutability, and accountability. Blockchain is slowly but surely set to radically change the mining paradigm.

Turn insight into action, make better decisions, and transform your business. Learn how.

 

 

 

Comments

Indranil Som

About Indranil Som

Indranil Som is the Digital Leader for Energy and Natural Resources industry at SAP India, engaged in consulting with C-level executives to enable organizations unlock business value through technology driven business transformations. He has had over 16 years of management consulting experience with a combination of strategy and technology engagements, encompassing scoping, planning and execution, with leading international firms.

Diving Deep Into Digital Experiences

Kai Goerlich

 

Google Cardboard VR goggles cost US$8
By 2019, immersive solutions
will be adopted in 20% of enterprise businesses
By 2025, the market for immersive hardware and software technology could be $182 billion
In 2017, Lowe’s launched
Holoroom How To VR DIY clinics

Link to Sources


From Dipping a Toe to Fully Immersed

The first wave of virtual reality (VR) and augmented reality (AR) is here,

using smartphones, glasses, and goggles to place us in the middle of 360-degree digital environments or overlay digital artifacts on the physical world. Prototypes, pilot projects, and first movers have already emerged:

  • Guiding warehouse pickers, cargo loaders, and truck drivers with AR
  • Overlaying constantly updated blueprints, measurements, and other construction data on building sites in real time with AR
  • Building 3D machine prototypes in VR for virtual testing and maintenance planning
  • Exhibiting new appliances and fixtures in a VR mockup of the customer’s home
  • Teaching medicine with AR tools that overlay diagnostics and instructions on patients’ bodies

A Vast Sea of Possibilities

Immersive technologies leapt forward in spring 2017 with the introduction of three new products:

  • Nvidia’s Project Holodeck, which generates shared photorealistic VR environments
  • A cloud-based platform for industrial AR from Lenovo New Vision AR and Wikitude
  • A workspace and headset from Meta that lets users use their hands to interact with AR artifacts

The Truly Digital Workplace

New immersive experiences won’t simply be new tools for existing tasks. They promise to create entirely new ways of working.

VR avatars that look and sound like their owners will soon be able to meet in realistic virtual meeting spaces without requiring users to leave their desks or even their homes. With enough computing power and a smart-enough AI, we could soon let VR avatars act as our proxies while we’re doing other things—and (theoretically) do it well enough that no one can tell the difference.

We’ll need a way to signal when an avatar is being human driven in real time, when it’s on autopilot, and when it’s owned by a bot.


What Is Immersion?

A completely immersive experience that’s indistinguishable from real life is impossible given the current constraints on power, throughput, and battery life.

To make current digital experiences more convincing, we’ll need interactive sensors in objects and materials, more powerful infrastructure to create realistic images, and smarter interfaces to interpret and interact with data.

When everything around us is intelligent and interactive, every environment could have an AR overlay or VR presence, with use cases ranging from gaming to firefighting.

We could see a backlash touting the superiority of the unmediated physical world—but multisensory immersive experiences that we can navigate in 360-degree space will change what we consider “real.”


Download the executive brief Diving Deep Into Digital Experiences.


Read the full article Swimming in the Immersive Digital Experience.

Comments

Kai Goerlich

About Kai Goerlich

Kai Goerlich is the Chief Futurist at SAP Innovation Center network His specialties include Competitive Intelligence, Market Intelligence, Corporate Foresight, Trends, Futuring and ideation. Share your thoughts with Kai on Twitter @KaiGoe.heif Futu

Tags:

Blockchain: Much Ado About Nothing? How Very Wrong!

Juergen Roehricht

Let me start with a quote from McKinsey, that in my view hits the nail right on the head:

“No matter what the context, there’s a strong possibility that blockchain will affect your business. The very big question is when.”

Now, in the industries that I cover in my role as general manager and innovation lead for travel and transportation/cargo, engineering, construction and operations, professional services, and media, I engage with many different digital leaders on a regular basis. We are having visionary conversations about the impact of digital technologies and digital transformation on business models and business processes and the way companies address them. Many topics are at different stages of the hype cycle, but the one that definitely stands out is blockchain as a new enabling technology in the enterprise space.

Just a few weeks ago, a customer said to me: “My board is all about blockchain, but I don’t get what the excitement is about – isn’t this just about Bitcoin and a cryptocurrency?”

I can totally understand his confusion. I’ve been talking to many blockchain experts who know that it will have a big impact on many industries and the related business communities. But even they are uncertain about the where, how, and when, and about the strategy on how to deal with it. The reason is that we often look at it from a technology point of view. This is a common mistake, as the starting point should be the business problem and the business issue or process that you want to solve or create.

In my many interactions with Torsten Zube, vice president and blockchain lead at the SAP Innovation Center Network (ICN) in Potsdam, Germany, he has made it very clear that it’s mandatory to “start by identifying the real business problem and then … figure out how blockchain can add value.” This is the right approach.

What we really need to do is provide guidance for our customers to enable them to bring this into the context of their business in order to understand and define valuable use cases for blockchain. We need to use design thinking or other creative strategies to identify the relevant fields for a particular company. We must work with our customers and review their processes and business models to determine which key blockchain aspects, such as provenance and trust, are crucial elements in their industry. This way, we can identify use cases in which blockchain will benefit their business and make their company more successful.

My highly regarded colleague Ulrich Scholl, who is responsible for externalizing the latest industry innovations, especially blockchain, in our SAP Industries organization, recently said: “These kinds of use cases are often not evident, as blockchain capabilities sometimes provide minor but crucial elements when used in combination with other enabling technologies such as IoT and machine learning.” In one recent and very interesting customer case from the autonomous province of South Tyrol, Italy, blockchain was one of various cloud platform services required to make this scenario happen.

How to identify “blockchainable” processes and business topics (value drivers)

To understand the true value and impact of blockchain, we need to keep in mind that a verified transaction can involve any kind of digital asset such as cryptocurrency, contracts, and records (for instance, assets can be tangible equipment or digital media). While blockchain can be used for many different scenarios, some don’t need blockchain technology because they could be handled by a simple ledger, managed and owned by the company, or have such a large volume of data that a distributed ledger cannot support it. Blockchain would not the right solution for these scenarios.

Here are some common factors that can help identify potential blockchain use cases:

  • Multiparty collaboration: Are many different parties, and not just one, involved in the process or scenario, but one party dominates everything? For example, a company with many parties in the ecosystem that are all connected to it but not in a network or more decentralized structure.
  • Process optimization: Will blockchain massively improve a process that today is performed manually, involves multiple parties, needs to be digitized, and is very cumbersome to manage or be part of?
  • Transparency and auditability: Is it important to offer each party transparency (e.g., on the origin, delivery, geolocation, and hand-overs) and auditable steps? (e.g., How can I be sure that the wine in my bottle really is from Bordeaux?)
  • Risk and fraud minimization: Does it help (or is there a need) to minimize risk and fraud for each party, or at least for most of them in the chain? (e.g., A company might want to know if its goods have suffered any shocks in transit or whether the predefined route was not followed.)

Connecting blockchain with the Internet of Things

This is where blockchain’s value can be increased and automated. Just think about a blockchain that is not just maintained or simply added by a human, but automatically acquires different signals from sensors, such as geolocation, temperature, shock, usage hours, alerts, etc. One that knows when a payment or any kind of money transfer has been made, a delivery has been received or arrived at its destination, or a digital asset has been downloaded from the Internet. The relevant automated actions or signals are then recorded in the distributed ledger/blockchain.

Of course, given the massive amount of data that is created by those sensors, automated signals, and data streams, it is imperative that only the very few pieces of data coming from a signal that are relevant for a specific business process or transaction be stored in a blockchain. By recording non-relevant data in a blockchain, we would soon hit data size and performance issues.

Ideas to ignite thinking in specific industries

  • The digital, “blockchained” physical asset (asset lifecycle management): No matter whether you build, use, or maintain an asset, such as a machine, a piece of equipment, a turbine, or a whole aircraft, a blockchain transaction (genesis block) can be created when the asset is created. The blockchain will contain all the contracts and information for the asset as a whole and its parts. In this scenario, an entry is made in the blockchain every time an asset is: sold; maintained by the producer or owner’s maintenance team; audited by a third-party auditor; has malfunctioning parts; sends or receives information from sensors; meets specific thresholds; has spare parts built in; requires a change to the purpose or the capability of the assets due to age or usage duration; receives (or doesn’t receive) payments; etc.
  • The delivery chain, bill of lading: In today’s world, shipping freight from A to B involves lots of manual steps. For example, a carrier receives a booking from a shipper or forwarder, confirms it, and, before the document cut-off time, receives the shipping instructions describing the content and how the master bill of lading should be created. The carrier creates the original bill of lading and hands it over to the ordering party (the current owner of the cargo). Today, that original paper-based bill of lading is required for the freight (the container) to be picked up at the destination (the port of discharge). Imagine if we could do this as a blockchain transaction and by forwarding a PDF by email. There would be one transaction at the beginning, when the shipping carrier creates the bill of lading. Then there would be look-ups, e.g., by the import and release processing clerk of the shipper at the port of discharge and the new owner of the cargo at the destination. Then another transaction could document that the container had been handed over.

The future

I personally believe in the massive transformative power of blockchain, even though we are just at the very beginning. This transformation will be achieved by looking at larger networks with many participants that all have a nearly equal part in a process. Today, many blockchain ideas still have a more centralistic approach, in which one company has a more prominent role than the (many) others and often is “managing” this blockchain/distributed ledger-supported process/approach.

But think about the delivery scenario today, where goods are shipped from one door or company to another door or company, across many parties in the delivery chain: from the shipper/producer via the third-party logistics service provider and/or freight forwarder; to the companies doing the actual transport, like vessels, trucks, aircraft, trains, cars, ferries, and so on; to the final destination/receiver. And all of this happens across many countries, many borders, many handovers, customs, etc., and involves a lot of paperwork, across all constituents.

“Blockchaining” this will be truly transformational. But it will need all constituents in the process or network to participate, even if they have different interests, and to agree on basic principles and an approach.

As Torsten Zube put it, I am not a “blockchain extremist” nor a denier that believes this is just a hype, but a realist open to embracing a new technology in order to change our processes for our collective benefit.

Turn insight into action, make better decisions, and transform your business. Learn how.

Comments

Juergen Roehricht

About Juergen Roehricht

Juergen Roehricht is General Manager of Services Industries and Innovation Lead of the Middle and Eastern Europe region for SAP. The industries he covers include travel and transportation; professional services; media; and engineering, construction and operations. Besides managing the business in those segments, Juergen is focused on supporting innovation and digital transformation strategies of SAP customers. With more than 20 years of experience in IT, he stays up to date on the leading edge of innovation, pioneering and bringing new technologies to market and providing thought leadership. He has published several articles and books, including Collaborative Business and The Multi-Channel Company.