Should I trust an AI's advice about how to spend my day?

PursueUpdated 17 October 20266 sources

Short answer

Trust it as far as it has earned, and no further. Across six experiments, people followed advice more readily when they believed it came from an algorithm rather than a person (Logg et al., 2019), yet watching an automated aid make mistakes made them distrust even a reliable one (Dzindolet et al., 2003).

01Too much and too little

Research on automation names two opposite errors: relying on a system too much (misuse) and ignoring it, often after false alarms (disuse)[1]. Because people can't fully understand complex systems, trust is what decides how much they rely on them, so the goal is trust that fits the system's real ability[2].

Across six experiments, ordinary people followed advice more when they thought it came from an algorithm rather than a person[3]. In other experiments, people expected an automated aid to beat them, then distrusted even a reliable aid after watching it make mistakes, unless they were told why it might err[4].

02Keeping your own judgment in charge

Designs that make people think before they see the AI's answer reduced over-reliance more than explanations did, though people liked them least[5]. Explanations help when a result disappoints, but too much explanation erodes trust again[6].

03Where the evidence stops

Much of this research comes from aviation, industry and estimation tasks, not from personal decisions about your own day. The direction is consistent; the size of each effect in daily life is not known.

Something to try today

Before you accept an assistant's suggestion for your day, say your own answer first, even in one word. Then compare. It is the cheapest version of the design that reduced over-reliance.

Pursue

How Tre-Up approaches this

Pursue sometimes asks one question before it suggests anything, so your judgment comes first. When it is unsure, it says why in one plain sentence.

Tre-Up is an iOS app in development. Pursue is the AI inside it. This note describes research and design intent, not results from the app. Nothing here is medical advice.

Sources

Summaries are written in our own words from each source's abstract, public record or official text. Follow the link to read the original. How we write these notes

  1. 1.Parasuraman R, Riley V (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 39(2):230–253.Catholic University of America · Honeywell Technology CenterOur summarydoi:10.1518/001872097778543886
  2. 2.Lee JD, See KA (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors 46(1):50–80.University of IowaOur summarydoi:10.1518/hfes.46.1.50_30392
  3. 3.Logg JM, Minson JA, Moore DA (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes 151:90–103.Harvard University · University of California, BerkeleyOur summarydoi:10.1016/j.obhdp.2018.12.005
  4. 4.Dzindolet MT, Peterson SA, Pomranky RA, Pierce LG, Beck HP (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies 58(6):697–718.Affiliation not yet confirmedOur summarydoi:10.1016/S1071-5819(03)00038-7
  5. 5.Buçinca Z, Malaya MB, Gajos KZ (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction 5(CSCW1).Harvard UniversityOur summarydoi:10.1145/3449287
  6. 6.Kizilcec RF (2016). How Much Information? Effects of Transparency on Trust in an Algorithmic Interface. Proceedings of CHI 2016, 2390–2395.Stanford UniversityOur summarydoi:10.1145/2858036.2858402

Keep reading

All notes
  1. 01How should an AI say it isn't sure?Plainly, and without inventing a confidence number. Modern neural networks are often over-confident (Guo et al., 2017), and a system allowed to decline uncertain cases makes far fewer errors on the ones it answers (Geifman and El-Yaniv, 2017); a 2019 review led from the University of Cambridge found that admitting uncertainty does not necessarily cost trust.Trust and uncertainty
  2. 02Why do I push back when an app tells me what to do?Because being pushed feels like losing a freedom, and people act to win it back; psychologists call this reactance. In an experiment with 394 people (Richards et al., 2022), simply reminding them the choice was theirs reduced the pushback against health messages, and the effect was still there a week later.Autonomy and wording
  3. 03How can an app help me if it doesn't know me yet?By asking, briefly, instead of guessing. A 2023 systematic review (Yuan and Hernandez) treats the 'cold start', a new user with no history, as its own problem, and asking the person directly is one approach that works from the first day. Later, fresh answers should outweigh old patterns, because preferences drift (Koren, 2009).Learning a person

Tre-Up is built on this research.

Pursue, the AI inside Tre-Up, is designed to notice when your day starts to slip and offer one small next step. The iOS app is in development. Invitations go out in small groups; leave your email and we will contact you when yours is ready.

Your email is used only for your invitation and important product updates. It is never sold. Unsubscribe at any time. Privacy policy