The use of artificial intelligence (AI) has gained considerable momentum recently, especially with the release of OpenAI’s chatbot ChatGTP1 AI has become more accessible, and it is now transforming the way people work, communicate and make decisions. Once again, it seems to have created a moment of technological disruption. However, even though AI is now part of everyday life, often there is no common understanding of what AI means. Perceptions of what AI is range from viewing it as a collection of fancy but harmless algorithms to considering it the end of humanity. In addition, there are very different opinions on whether AI possesses more or fewer desired qualities such as fairness, accuracy, and robustness than humans or traditional rule-based software.
However, it is obvious that what is not obvious is how to deal with AI. How much trust can be put into the output of ChatGPT? Can organizations rely on AI for making hiring decisions? What should people think of an AI tool determining their credit limit? The lack of clarity around the use of AI has urged regulators to start putting forth proposals for legal frameworks. In particular, the EU AI Act, that aims to regulate the ethical and safe deployment of AI in the European Union, might soon be implemented. In December 2022, the European Council reached a consensus on a negotiating mandate regarding the proposal for the EU AI Act.2
Next to risk that concerns many deployed IT systems, such as a lack of security and data protection, there is also specific risk that is unique to AI systems (i.e., AI based on machine learning). Currently, there are no established techniques available to assess and mitigate this risk. Thus, it is crucial to understand what makes AI so different from traditional rule-based systems and how to address these differences. There are 3 characteristics of AI systems that are particularly relevant when comparing rule-based systems to AI: explainability, fairness and robustness.
It is crucial to understand what makes AI so different from traditional rule-based systems and how to address these differences.Explainability
An AI system is explainable if its output and the reasoning behind it can be understood and trusted by a human. Not being able to explain the outputs of an AI system makes it difficult to control its correctness and lack of bias. Where rule-based algorithms can be understood and explained based on predefined rules, many AI models are considered so-called black boxes. In fact, the inner workings of some of the more complex models are hard to understand even for their developers. As AI is used more pervasively, and for critical purposes such as medical diagnoses or hiring decisions, explainability will become indispensable. To meet the requirements of explainability, one option is to use very simple AI models that can be easily interpreted. However, for many use cases, such simple models are not sufficient. For the more complex models, researchers and practitioners are developing both model-agnostic and model-specific methods. In particular, big tech enterprises such as IBM, Microsoft and Google provide tools for so-called explainable AI (XAI): AI Explainability 3603 and InterpretML.4
Fairness
Recently, there have been outcries from people feeling discriminated against by AI systems; for instance, by an employment agency in Austria.5 Having fairness in an AI system means ensuring its neutrality and preventing discrimination based on protected attributes such as race, gender and age. Since AI is trained on real-world data, it mirrors the biases it learns from these data (e.g., social, historical biases). In that sense, AI cannot be accused of being more biased than the society by which its training data are generated. A rule-based system is not necessarily free of bias either because its developers might have unconsciously infused their biases into the system. However, since AI systems can operate at scale (e.g., ChatGPT) and affect large populations, they can systematically perpetuate and amplify biases and discriminatory patterns. To mitigate unfairness in AI models, researchers and practitioners are working to identify bias through fairness metrics and mitigate it with bias correction algorithms. However, choosing the right fairness metrics is far from trivial because the interpretation of fairness is subjective and varies among individuals. Indeed, fairness considerations for AI involve substantial ethical reflection before translating a common understanding of fairness into technical requirements. Thus, the fairness of AI models is not only a technical question, but also very much an ethical question and it calls for corresponding expertise.
Robustness
The robustness of an AI model refers to its ability to withstand the challenges of a new environment and unforeseen inputs. For example, an AI model can suffer from a lack of robustness if it learned too closely from the training data (overfitting) or the training data were statistically different from the data the model is tested on when deployed (data distribution shift). AI systems are also at risk of malicious actors exploiting vulnerabilities specific to the model by introducing corrupt or misleading data into the training data set (data poisoning) or tricking the system with perturbed data (adversarial attacks). For example, an image recognition system might categorize the image of an apple as “apple,” but if an attacker puts a sticker on the apple saying “iPod,” it might categorize the image as an iPod.6 Robustness is also decreased by so-called short-cut learning where an AI model does not grasp underlying concepts in the data and only learns superficial features. Consequently, for example, an image classification model might categorize an object correctly most of the time, but not if it is presented in an uncommon way, such as upside down. Compared to rule-based systems, it is more difficult to validate AI models. Enhancing the robustness of AI models can be accomplished through thorough testing in diverse scenarios and subsequent monitoring of the model for performance drift. In addition, training the model on representative data in the first place helps to make it more robust.
Risk Assessment
Explainability, fairness, and robustness are only a selection of characteristics required of AI systems. Many more need to be addressed to fully evaluate an AI-specific risk profile. To help avoid blind spots, frameworks such as the Eraneos AuditingAI Framework (figure 1) can be applied. This framework is based on experiences in auditing AI use cases and research into AI failures. It distinguishes between the perspective of an organization using an AI system and that of a specific use case. Based on the questions for risk assessment and risk mitigation measures that the framework defines, governance and AI-specific risk areas across the entire life cycle of the system can be evaluated.
Figure 1—Eraneos AuditingAI Framework
Compared to classic IT frameworks, the Eraneos AuditingAI framework is tailored to AI and rather explorative. For many of the defined characteristics such as explainability and robustness, there are no established controls. New requirements might emerge for new AI models, in which casethe framework will have to be adapted. For example, there are ongoing debates regarding the truthfulness of ChatGPT.7 Although the framework can be applied to any AI model, it is important to consider that some of the use-case specific risk areas are more relevant than others depending on the type of the model. For example, explainability can be easily addressed for a linear regression model, but for a voice recognition system based on deep-learning models (i.e., very complex models) it is more difficult to address. In this regard, it is crucial to conduct a risk assessment and identify the relevant risk areas for any AI system before conducting an audit.
Conclusion
As AI models become more sophisticated and more widely adopted across various industries, risk mitigation strategies need to be continuously adapted. For now, it is important to keep in mind:
- As AI is disrupting everyday lives and work and regulations are on their way, now is the time to prepare for risk assessments and mitigation.
- The risk profile of AI is different from that of traditional rule-based software. In particular, in AI systems, explainability, fairness and robustness are necessary.
- To avoid blind spots during risk assessment, explorative frameworks can be used.
Endnotes
1 OpenAI, “Introducing ChatGPT,” 30 November 2022
2 European Council of the European Union, “Transport, Telecommunications and Energy Council (Telecommunications),” 6 December 2022
3 IBM Research Trusted AI, “AI Explainability 360”
4 InterpretML
5 Kayser-Bril, N.; “Austria’s Employment Agency Rolls Out Discriminatory Algorithm, Sees No Problem,” Algorithm Watch, 6 October 2019
6 OpenAI, “Multimodal Neurons in Artificial Neural Networks,” 4 March 2021
7 Sobieszek, A.; T. Price; “Playing Games With AIs: The Limits of GPT-3 and Similar Large Language Models,” Minds and Machines, vol. 32, iss. 2, p. 341–364
Belinda Mueller
Is a consultant at Eraneos Group where she advises clients on trustworthy artificial intelligence (AI). She has a background in linguistics, cognitive science and statistics, and thus a good understanding of the mechanisms of AI and its impact on society. Because she has worked as data scientist at ELCA Informatik AG, Mueller brings practical experience in machine learning and data analysis to her consulting work. Also, she regularly speaks at events and conferences, sharing her learnings on the responsible use of AI.