Can a routing threshold keep a large model off routine cases while sending it the decisions that need another read?
An encoder makes the first prediction. Moving a confidence threshold changes the fraction of cases a second model sees. In 100 constructed example cases, a strict threshold sends 12 to the second model, a middle one sends 29, and a loose one sends 57.
What the authors tested
A small encoder makes the first prediction on every case. A confidence threshold decides which cases also go to a larger second model. The authors moved the threshold through three settings and counted how many of 100 example cases reached the second model at each.
Figure 1. Cases sent for a second read
Text alternative: a strict threshold sends 12 of 100 cases to the second model, a middle threshold sends 29, and a loose threshold sends 57.Looser thresholds send more cases to the second model. Counts are out of 100 example cases.Source: constructed values for this illustrative page. Not measured data.
Limitations
The trade-off depends on how well confidence holds up on new data. The illustrated routing counts do not establish accuracy on future traffic.
Treat the saving as a measurement on these example cases, and test it on your own traffic before using it in a deployment plan.
Cite this example
Illustrative record. There is no DOI; the identifier is assigned on publication.
Vale, M. "When should a small model ask for help?" TenPeers, The inference issue (example issue, illustrative; not a published paper).
Identifier: assigned on publication.
Cohort: 100 eligible papers (example). Expected impact 8th of 100; talk interest 2nd of 100.