Get your AI ready for safety-critical applications

© Fraunhofer-Gesellschaft

Production planning, logistics, maintenance, quality control – in industrial manufacturing, there are many areas where artificial intelligence is used. In practice, however, AI models are still rarely utilized. The reason: reliability is difficult to verify. New certification criteria can make AI suitable for safety-critical applications.

The expectations are hardly to be topped: artificial intelligence should make production more flexible, plan maintenance proactively, optimize the flow of goods, automate logistics, and automate quality controls. "In fact, numerous promising AI algorithms and architectures have been developed in recent years – including at the Fraunhofer IPA – for computer vision, human-machine interfaces, or networked robotics," reports Xinyang Wu from the Center for Cyber Cognitive Intelligence at IPA. What is now missing is practical implementation. "There is a gap between research and application. In industry, the new AI applications are only slowly being adopted. They are considered not reliable enough for safety-critical applications."

Wu knows the reservations of users firsthand: "When we speak with our partners from industry, it quickly becomes clear that companies only want to use autonomous and self-learning robots if they work absolutely reliably, and if one can say with 100% certainty that the machines pose no danger to humans."

Exactly that cannot currently be proven. There are neither standards nor standardized tests. However, these are urgently needed, emphasizes Wu: "The goal must be to certify and make transparent the decisions made by algorithms. For example, traceability must be guaranteed: if a machine makes decisions independently, I must – at least in retrospect – be able to find out why it made a mistake in a particular situation. Only then can it be prevented from happening again. Black-box models, where the decision of the algorithms cannot be traced, are not suitable for safety-critical applications unless the model is certified using the correct method, in our opinion."

But how do you verify artificial intelligence? The IPA team at the Center for Cyber Cognitive Intelligence has now proposed a strategy and reports on the state of the art in the white paper "Reliable AI – Deploying AI in safety-critical industrial applications": The strategy is based on certifiability and transparency.

Criteria Catalog for Greater Safety

"Primarily, our goal was to find rules that can be used to evaluate the reliability of machine learning and associated AI," reports Wu. The result of this research is five criteria that AI systems should meet to be considered safe:

- All decisions made by algorithms must be understandable to humans.
- The function of the algorithms must be verified before deployment using methods of formal verification.
- Additionally, statistical validation is necessary, especially when formal verification is not feasible due to scalability issues for the specific application. This can be checked through test runs with larger datasets or batch sizes.
- The uncertainties underlying the decisions of neural networks must also be identified and quantified.
- During operation, systems must be continuously monitored, for example through online monitoring. It is important to record input and output – that is, sensor data and the decisions resulting from their analysis.

The five criteria could form the basis for a future standardized testing process, emphasizes Wu: "At IPA, we have already compiled different algorithms and methods for each of these points, which can actually be used to verify the reliability of AI systems. We have also conducted such tests with some of our customers."

Transparency Builds Trust

The second fundamental prerequisite for the safe use of AI systems is their transparency. According to the ethical guidelines of the "High-Level Expert Group on Artificial Intelligence" of the European Commission, abbreviated HLEG AI, this is one of the key elements for the realization of trustworthy AI. Unlike the criteria used to verify reliability at the algorithmic level, this transparency relates solely to the interaction with humans at the systematic level. Three points are summarized from the HLEG AI guidelines that transparent AI must fulfill: First, decisions made by algorithms must be understandable. Second, it must be possible for humans to explain these decisions on a comprehensive level of human understanding.

And third, AI systems must communicate with humans and inform them about the capabilities of the algorithms and where their limits lie.

"Only if it is possible to test the reliability of self-learning, autonomous AI systems with standardized procedures and also consider ethical aspects will AI users trust them – whether in road traffic or in factory halls," predicts Wu. "When this trust is established, the gap between research and application will close."

The study "Reliable AI – Deploying AI in safety-critical industrial applications" is available for download at: https://www.ki-fortschrittszentrum.de/de/studien/zuverlaessige-ki.html