Topic 05
Trust and uncertainty
How people come to rely on automated advice, and how a system should admit what it doesn't know.
People rely on automation too much or too little, and often follow an algorithm's advice more readily than a person's. One visible mistake can then make them distrust even a reliable system, unless they know why it might err.
Machine-learning research shows systems are often over-confident, and that declining uncertain cases makes the remaining answers more reliable. The goal is trust that matches what a system can really do.
Notes
Questions in this topic
- 01Should I trust an AI's advice about how to spend my day?Trust it as far as it has earned, and no further. Across six experiments, people followed advice more readily when they believed it came from an algorithm rather than a person (Logg et al., 2019), yet watching an automated aid make mistakes made them distrust even a reliable one (Dzindolet et al., 2003).Trust and uncertainty
- 02How should an AI say it isn't sure?Plainly, and without inventing a confidence number. Modern neural networks are often over-confident (Guo et al., 2017), and a system allowed to decline uncertain cases makes far fewer errors on the ones it answers (Geifman and El-Yaniv, 2017); a 2019 review led from the University of Cambridge found that admitting uncertainty does not necessarily cost trust.Trust and uncertainty
10 sources
What this topic rests on
- 1999Principles of mixed-initiative user interfacesHorvitz E · Proceedings of CHI 1999, 159–166Microsoft ResearchDesign principles
- 2004Trust in Automation: Designing for Appropriate RelianceLee JD, See KA · Human Factors 46(1):50–80University of IowaReview
- 1997Humans and Automation: Use, Misuse, Disuse, AbuseParasuraman R, Riley V · Human Factors 39(2):230–253Catholic University of America · Honeywell Technology CenterReview
- 2003The role of trust in automation relianceDzindolet MT, Peterson SA, Pomranky RA, Pierce LG, Beck HP · International Journal of Human-Computer Studies 58(6):697–718Affiliation not yet confirmedLab experiments
- 2019Algorithm appreciation: People prefer algorithmic to human judgmentLogg JM, Minson JA, Moore DA · Organizational Behavior and Human Decision Processes 151:90–103Harvard University · University of California, BerkeleySix experiments
- 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingBuçinca Z, Malaya MB, Gajos KZ · Proceedings of the ACM on Human-Computer Interaction 5(CSCW1)Harvard UniversityOnline experiment
- 2016How Much Information? Effects of Transparency on Trust in an Algorithmic InterfaceKizilcec RF · Proceedings of CHI 2016, 2390–2395Stanford UniversityOnline field experiment
- 2019Communicating uncertainty about facts, numbers and sciencevan der Bles AM, van der Linden S, Freeman ALJ, Mitchell J, Galvao AB, Zaval L, Spiegelhalter DJ · Royal Society Open Science 6(5):181870University of Cambridge · University of Warwick · Columbia UniversityInterdisciplinary review
- 2017On Calibration of Modern Neural NetworksGuo C, Pleiss G, Sun Y, Weinberger KQ · Proceedings of ICML 2017, PMLR 70:1321–1330Cornell UniversityMachine-learning experiments
- 2017Selective Classification for Deep Neural NetworksGeifman Y, El-Yaniv R · Advances in Neural Information Processing Systems 30 (NeurIPS 2017)Technion – Israel Institute of TechnologyMachine-learning method and experiments
Tre-Up is built on this research.
Pursue, the AI inside Tre-Up, is designed to notice when your day starts to slip and offer one small next step. The iOS app is in development. Invitations go out in small groups; leave your email and we will contact you when yours is ready.
Your email is used only for your invitation and important product updates. It is never sold. Unsubscribe at any time. Privacy policy