Machine-learning method and experiments · 2017
Selective Classification for Deep Neural Networks
Geifman Y, El-Yaniv R
What it found
A trained network can be given the option to decline when it is unsure, set so that its error rate stays under a chosen level. Answering fewer cases made the answers it did give far more reliable.
Where the evidence stops
Image classification benchmarks; the mapping to personal advice is an analogy.
How it shapes Pursue
Pursue
Declining to answer is a legitimate action. Pursue can say nothing, or ask, instead of guessing.
A design input, not evidence that Pursue works.
Cited in
- 01How should an AI say it isn't sure?Plainly, and without inventing a confidence number. Modern neural networks are often over-confident (Guo et al., 2017), and a system allowed to decline uncertain cases makes far fewer errors on the ones it answers (Geifman and El-Yaniv, 2017); a 2019 review led from the University of Cambridge found that admitting uncertainty does not necessarily cost trust.Trust and uncertainty
- 02When is it better for an AI assistant to say nothing?Whenever speaking would cost more attention than it gives back: when the person can't take it in, the evidence is thin, or nothing needs to change. The 2018 framework for just-in-time support (Nahum-Shani et al.) counts giving nothing as a designed option, and declining uncertain cases makes a system's other answers more reliable (Geifman and El-Yaniv, 2017).Timing and receptivity