Can AI be relied upon to run mission-critical systems and make judgement calls?
Concerns are on the rise about reliability in AI predictions. Human intuition still beats AI hands down in making judgment calls in a crisis. People – especially those working in their areas of expertise – are simply more trustworthy. Studies have shown that professionals such as air traffic controllers or nuclear power plant operators are highly reliable even in high-risk situations as, unlike AI systems, they can detect, contain and recover from errors, and practice improvisational problem-solving. While current AI systems are great at situational awareness, they are less good at anomaly detection, and improvising solutions. Can AI mature? asks Fred Werner, Head of Strategic Engagement, ITU Telecommunication Standardisation Bureau.
As artificial systems (AI) get increasingly complex, they are being used to make forecasts – or rather generate predictive model results – in more and more areas of our lives. But, at the same time, concerns are on the rise about reliability, amid widening margins of error in elaborate AI predictions.
How can we address these concerns?
Management science offers a set of tools that can make AI systems more trustworthy, according to Thomas G Dietterich, Professor Emeritus and Director of Intelligent Systems Research at Oregon State University. During a webinar on the AI for Good platform hosted by the International Telecommunication Union (ITU), he told our audience that the discipline that brings human decision-makers to the top of their game can also be applied to machines
Why is this important? Because human intuition still beats AI hands down in making judgment calls in a crisis. People – and especially those working in their areas of experience and expertise – are simply more trustworthy. Studies by the University of California (UC), Berkeley, scholars Todd LaPorte, Gene Rochlin and Karlene Roberts found that certain groups of professionals, such as air traffic controllers or nuclear power plant operators, are highly reliable even in a high-risk situation.
These professionals develop a capability to detect, contain and recover from errors, and practice improvisational problem solving, said Dietterich. This is because of their “preoccupation with failure”. They are constantly watching for anomalies and near misses – and treating those as symptoms of a potential failure mode in the system. Anomalies and near misses, rather than being brushed aside, are then studied for possible explanations, normally by a diverse team with wide-ranging specializations. Human professionals bring far higher levels of “situational awareness” and know when to defer to each other’s expertise.
These principles are useful when thinking about how to build an entirely autonomous and reliable AI system, or how to design ways for human organizations and AI systems to work together. AI systems can acquire high situational awareness, thanks to their ability to integrate data from multiple sources and continually re-assess risks. However current AI systems, while adept at situational awareness, are less effective at anomaly detection and unable to explain anomalies and improvise solutions.
More research is needed before an AI system can reliably identify and explain near-misses. We have systems that can diagnose known failures, but how do we diagnose unknown failures? What would it mean for an AI system to engage in improvisational problem solving that somehow can extend the space of possibilities beyond the initial problem that the system was programmed to solve?
Where AI systems and humans collaborate, a shared mental model is needed. AI should not bombard its human counterparts with irrelevant information, and must also understand and be able to predict the behaviour of human teams.
One way to train machines to explain anomalies, or to deal with spontaneity, could be exposure to the performing arts. Researchers and musicians at the Monash University in Melbourne and Goldsmiths University of London set out to explore whether AI could perform as an improvising musician in a phantom jam session. Mark d’Inverno, a jazz pianist and Professor of Computer Science at Goldsmiths in London, improvised live with Melbourne-based drummer and Monash University researcher Alon Ilsar. Completing the trio was an AI system, participating as a musician as well as an intermediary for the two artists who had never played together before. During the session, the notes played on a MIDI piano in London by d’Inverno fed an algorithm, which modelled them to generate new notes in real-time and transmit them to Ilsar in Melbourne. Ilsar improvised in response with an AirSticks gestural instrument for electronic percussion.
Their goal was to emulate the real-life process of improvisation. Free-flowing, spontaneous improvisations are often considered the truest expression of creative artistic collaboration among musicians. ‘Jamming’ not only requires musical ability, but also trust, intuition and empathy towards one’s bandmates.
In the study, the first setting, called ‘Parrot’, repeats whatever is played. The second system autonomously plays notes regardless of a human musician’s contribution. The third also features complete autonomy, but counts the number of notes being played by the human musician to define the energy of the music. The fourth and most complicated system builds a mathematical model of the human artist’s music. It listens carefully to what the musicians play and builds a statistical model of the notes, their patterns and even stores chord sequences.
Adding to this human/AI jamming session approach, Professor Dietterich see a further two promising approaches to improve, and mathematically “guarantee” trustworthiness. One is a competence model that can compute quantile regressions to predict AI behaviour, using the “conformal prediction” method to make additional corrections. Yet this approach requires lots of data and remains prone to misinterpretation.
The other way is to make autonomous systems deal with their “unknown unknowns” via open category detection. For instance, a self-driving car trained on European roads might have problems with kangaroos in Australia. An anomaly detector using unlabelled data could help the AI system respond more effectively to surprises.
READ MORE:
- The power of AI in accelerating growth
- Why developers are our best defence against cyberattacks
- Unified data drives the factory of tomorrow
- CTOs share advice to their 25-year-old selves
As AI is deployed in more and more areas of our lives, what is becoming clear is that, far from a nightmare scenario of the machines taking over, the only way AI can be made more reliable, and more effective is for there to be a tighter than ever symbiosis between human systems and AI systems. Only then can we truly rely on AI.
For more news from Top Business Tech, don’t forget to subscribe to our daily bulletin!
Follow us on LinkedIn and Twitter
