A Checklist Of Ethical Design Challenges For Business AI

Guido Wagner

In the first part of this blog, we saw that design principles and digital ethics build trust in artificial intelligence (AI). However, we didn’t address the areas where we are most likely to find those ethical challenges. A checklist can help identify the most relevant pitfalls. Teams should define such a list to ensure that they are properly and responsibly bringing an AI system into the world. Not only is it important to plan, implement, and operate an AI system carefully, but also to create a plan to terminate it if needed.

The following questions can help formulate such a checklist. Most of the items apply to multiple phases and are hopefully useful for teams in creating individualized, AI-specific checklists.

Human involvement

Let’s start with some questions about human involvement in AI-driven processes.

  • What user roles are involved and what are the individual ethical requirements of those users? (For example, a hotel reception clerk would have very different requirements than a doctor.)
  • In which contexts will the users work? (Create parameters for these contexts to help the AI “understand” the users’ current situation and adjust depending on their ethical needs. For example, if a purchasing process has the parameter “urgency,” it may be acceptable to not purchase from a fair-trade source if the purchase is deemed urgent.)
  • How is each user’s current state evaluated and how is that information used? (For example, if eye-tracking technology is used, what happens with the information that is gathered concerning the users’ reading behavior and emotional state?)
  • What is the user’s cultural environment? Which ethical and social values do the users share?
  • Will there be situations in which the user needs to or wants to completely or partly turn off the AI’s functions? Should you develop an “escape door” for the AI functions? How can the user be supported if such a situation should occur?
  • If explicit user feedback is requested, how will it be evaluated? (For example, if a system has the ability to learn from observing users when they analyze the reasons for a low-revenue month, does the AI need to ask the users for permission to be trained by observing their behavior?)

Algorithms and boundary conditions

One requirement of business AI is to be transparent, especially about the reasons for a result. This can be challenging (or even impossible) for some AI technologies like neural networks. Here are some checklist items concerning technological boundaries.

  • How can “learn and forget” processes be monitored (e.g., in case of outdated data or legal deletion obligations)? How will humans be involved here (e.g., in the form of an “AI auditor”)?
  • What are the rules for AI “learn & forget”? How can they be customized? Who can customize them?
  • How can the quality of information used for learning be monitored?
  • How will machines evaluate when a human must be involved in a decision or when to seek advice from a human?
  • What are the limits for decisions or proposals created by the system? (For example, the process of human-AI-interaction might depend on parameters like, “if the goods to be purchased are valued over 10,000 euros, then a human decision maker must be involved.”)
  • In which cases does a process require a four-eyes principle (i.e., two people must review everything) or other trust-building measures?
  • Can computational results – and how they were achieved – be understood by humans? Is it possible to understand them immediately or after certain actions? (For example, normally the user may not need an explanation of why an AI came to a decision. But, if the user wants an explanation, the system must be able to provide it.)
  • Can computational results be reproduced (without and after additional learning steps)? If not, how will the users be informed about this?
  • In which process steps are “escape doors” required or reasonable? To which extent should they stop AI functions? How can they be tested?
  • Is the data used for learning free of bias? If not, what are the sources of bias? Can the level of bias be analyzed and quantified?
  • To which extent does bias affect the system’s computational results?
  • How can the level of bias be shown to the user or other systems using the results?

Compliance and security of learning systems

Legal compliance and data security are strongly related to digital ethics. The following are some fundamental questions to consider.

  • Which legal compliance topics must be covered (including data privacy)? Which internal company policies are applicable? How can compliance be monitored during development and operations?
  • In terms of self-regulation of AI providers, are the AI development plans compliant with the company’s own standards?
  • Are there possible liability issues resulting from AI-made decisions? In which situations?
  • Would it be possible to intentionally teach the system the wrong things? How can security breaches (including incorrect teaching) be uncovered?
  • How can the implementation of backdoors be avoided and detected?
  • How can potential, hidden override directives be unveiled?
  • In which cases must the system’s life be ended? What happens with the learned information in such a case?

Potential long-term impact

According to Amara’s law, “we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.” Therefore, it makes sense to look at potential long-term ethical questions.

  • How can a long-term risk assessment be done? What will be the impact on society, if any, when tasks are automated on a large scale?
  • If the system is widely used, would humanity lose knowledge or capabilities?
  • How can behavioral changes of the AI system be detected?
  • How can unintended or irrational computational results be monitored – and be avoided or filtered?
  • How can we detect if the system starts working independently of any human-defined task? How will we react in such a case?

The most important checkpoint

Running through this checklist helps identify the requirements for a business system that acts in a sustainably ethical way. The most important checkpoint was not mentioned: The list must be reviewed and updated on a regular basis. And, as the level of AI intelligence grows, checklist items must become more granular.

Technological advances must be foreseen to avoid AI developing faster than people’s ability to define and implement an ethical foundation. We need things like bias handling, escape doors, and AI auditors. There are likely many more pitfalls and challenges out there, waiting to be seen before we can add a real ethical “conscience” to what is called artificial intelligence.

Highly skilled human employees will be vital during this transformation and beyond. Learn more about The Human Side Of Machine Learning.


Guido Wagner

About Guido Wagner

Guido Wagner is responsible for invention projects in SAP Design. He focuses on user experience optimization in a digitalized business environment. Preparing for the future of work through sustainable artificial intelligence that improves the way people live is his passion. Share your thoughts with Guido on LinkedIn.