Mastering Artificial Intelligence

© Fraunhofer IPA

© Fraunhofer IPA

Artificial intelligence often has a black-box character. But only transparency can build trust. To explain the respective solution process, there is specialized software. A study by the Fraunhofer IPA has now compared and evaluated different methods that make machine learning procedures explainable.

Artificial intelligence, which was science fiction just a few decades ago, has now become part of everyday life. In manufacturing, it detects anomalies in the production process; in banking, it decides on loans; and on Netflix, it finds the right film for every customer. Behind this are highly complex algorithms operating in the background. The more demanding the problem, the more complex the AI model—and thus also more opaque.

But users especially want to understand how a decision is made, particularly in critical applications: Why was the workpiece sorted out as defective? What causes the wear of my machine? Only in this way can improvements be made, which increasingly also concern safety. Additionally, the European General Data Protection Regulation (GDPR) requires decisions to be transparent and understandable.

Software comparison for xAI

To solve this problem, an entire research field has emerged: "Explainable Artificial Intelligence," or xAI for short. There are now numerous digital tools on the market that make complex AI solution pathways explainable. For example, they highlight in an image the pixels that led to the sorting out of defective parts. Experts from the Fraunhofer Institute for Production Technology and Automation IPA in Stuttgart have compared nine common explanation methods—such as LIME, SHAP, or Layer-Wise Relevance Propagation—and evaluated them using sample applications. Three criteria were particularly important:

- Stability: For the same task, the program should always provide the same explanation. It should not happen that an anomaly in the production machine is attributed to Sensor A once and Sensor B another time. This would destroy trust in the algorithm and hinder the derivation of actionable options.
- Consistency: At the same time, only slightly different input data should result in similar explanations.
- Fidelity: It is especially important that explanations truly reflect the behavior of the AI model. It should not happen that the explanation for denying a bank loan cites an age that is too high, although the actual reason was the insufficient income.

The application case is decisive

Conclusion of the study: All examined explanation methods proved to be useful. "But there is no one perfect method," says Nina Schaaf, who was responsible for the study at Fraunhofer IPA. There are significant differences, for example, in the runtime required by a method. The choice of the best software also depends heavily on the specific task. For instance, Layer-Wise Relevance Propagation and Integrated Gradients are particularly well suited for image data. "And finally, the target audience of an explanation is always important: an AI developer wants and should receive an explanation presented differently than the production manager, because both draw different conclusions from the explanations," Schaaf summarizes.

The study "Explainable AI in Practice" is available free of charge at the following link: https://www.ki-fortschrittszentrum.de/de/studien/erklaerbare-ki-in-der-praxis.html