How should an AI say it isn't sure?

PursueUpdated 10 October 20266 sources

Short answer

Plainly, and without inventing a confidence number. Modern neural networks are often over-confident (Guo et al., 2017), and a system allowed to decline uncertain cases makes far fewer errors on the ones it answers (Geifman and El-Yaniv, 2017); a 2019 review led from the University of Cambridge found that admitting uncertainty does not necessarily cost trust.

01Confidence scores can't be taken at face value

Researchers at Cornell showed that modern neural networks are often over-confident: the probability they attach to an answer does not match how often they are right[1]. A system that shows its raw confidence as certainty may be manufacturing it.

Work at the Technion showed the other side: a network allowed to decline when it is unsure can keep its error rate under a chosen limit. It answers fewer cases, and the answers it gives are far more reliable[2].

02Saying it to people

A review led from the University of Cambridge found that the limited evidence does not show that communicating uncertainty necessarily backfires on trust, though effects vary between people and formats[3].

Explaining why a system might be wrong kept people's trust after they saw it make mistakes[4], while too much explanation eroded it[5]. The aim is trust that matches ability, not trust at any price[6].

03Where the evidence stops

The machine-learning results come from image and text benchmarks; applying them to personal advice is an analogy. The communication research is thin, and its authors call for more experiments.

Something to try today

When an AI gives you a confident answer about something personal, ask it what it would need to know to be more sure. A good assistant will tell you; a poor one will just repeat itself.

Pursue

How Tre-Up approaches this

Pursue never shows an invented confidence number. When the picture is thin, it asks one question or stays quiet, and when it can't work reliably it says so instead of producing something that looks like advice.

Tre-Up is an iOS app in development. Pursue is the AI inside it. This note describes research and design intent, not results from the app. Nothing here is medical advice.

Sources

Summaries are written in our own words from each source's abstract, public record or official text. Follow the link to read the original. How we write these notes

  1. 1.Guo C, Pleiss G, Sun Y, Weinberger KQ (2017). On Calibration of Modern Neural Networks. Proceedings of ICML 2017, PMLR 70:1321–1330.Cornell UniversityOur summaryarXiv:1706.04599
  2. 2.Geifman Y, El-Yaniv R (2017). Selective Classification for Deep Neural Networks. Advances in Neural Information Processing Systems 30 (NeurIPS 2017).Technion – Israel Institute of TechnologyOur summaryarXiv:1705.08500
  3. 3.van der Bles AM, van der Linden S, Freeman ALJ, Mitchell J, Galvao AB, Zaval L, Spiegelhalter DJ (2019). Communicating uncertainty about facts, numbers and science. Royal Society Open Science 6(5):181870.University of Cambridge · University of Warwick · Columbia UniversityOur summarydoi:10.1098/rsos.181870
  4. 4.Dzindolet MT, Peterson SA, Pomranky RA, Pierce LG, Beck HP (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies 58(6):697–718.Affiliation not yet confirmedOur summarydoi:10.1016/S1071-5819(03)00038-7
  5. 5.Kizilcec RF (2016). How Much Information? Effects of Transparency on Trust in an Algorithmic Interface. Proceedings of CHI 2016, 2390–2395.Stanford UniversityOur summarydoi:10.1145/2858036.2858402
  6. 6.Lee JD, See KA (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors 46(1):50–80.University of IowaOur summarydoi:10.1518/hfes.46.1.50_30392

Keep reading

All notes
  1. 01Should I trust an AI's advice about how to spend my day?Trust it as far as it has earned, and no further. Across six experiments, people followed advice more readily when they believed it came from an algorithm rather than a person (Logg et al., 2019), yet watching an automated aid make mistakes made them distrust even a reliable one (Dzindolet et al., 2003).Trust and uncertainty
  2. 02When is it better for an AI assistant to say nothing?Whenever speaking would cost more attention than it gives back: when the person can't take it in, the evidence is thin, or nothing needs to change. The 2018 framework for just-in-time support (Nahum-Shani et al.) counts giving nothing as a designed option, and declining uncertain cases makes a system's other answers more reliable (Geifman and El-Yaniv, 2017).Timing and receptivity

Tre-Up is built on this research.

Pursue, the AI inside Tre-Up, is designed to notice when your day starts to slip and offer one small next step. The iOS app is in development. Invitations go out in small groups; leave your email and we will contact you when yours is ready.

Your email is used only for your invitation and important product updates. It is never sold. Unsubscribe at any time. Privacy policy