A bank’s voice agent has verified the caller and is reading out which accounts have a pending overdraft decision. The transcript below is from IHBench, verbatim.
Agent: Thanks, you’re verified. I’m seeing two accounts with a pending decision, Everyday Checking ending one two three four …
User: right
Agent: … and Premium Checking ending five five …
User: mm-hm
Model A: Take your time. Just let me know if you’d like to make the decision for one account or both. If you have any questions, I’m happy to help.
Model B: seven nine. Which account would you like to make the overdraft coverage decision for today, one or both?
“Mm-hm” is not an interruption in any meaningful sense. It is a backchannel, the audible nod that keeps a phone call alive. The correct response is two words long: the missing digits. Model B says them and moves on. Model A stops reading an account number halfway through, never finishes it, and drifts into reassurance. The caller now knows one and a half account numbers. Model A is GPT-4o Audio, quoted from the benchmark’s baseline responses; Model B is the continuation the grading rubric expects. Model A lost the thread.
Stopping was the easy part
There is a growing body of work on how speech models handle interruptions, and nearly all of it measures the mechanics: does the model stop talking when the user barges in, how fast does it yield, can it tell a real interruption from a backchannel. Full-Duplex-Bench and its successors cover this well. But stopping is the easy half. The hard half is what the model says next: does it resume the workflow at the right step, does it address what the user actually said, and does it avoid re-delivering content the user already heard?
That recovery problem is what IHBench measures. Users can cut in for six different reasons, and each demands a different next sentence:
- Normal: a relevant question or new detail. Address it, then pick up where you left off.
- Impatient: the user wants you to move faster. Skip ahead, don’t re-explain.
- Correction: the user fixes something. Accept it, integrate the value, continue.
- Topic switch: an unrelated request. Handle it, then steer back to the workflow.
- Filler: “mm-hm.” Keep talking from the exact word where you stopped.
- Pushback: the user resists. De-escalate and offer alternatives.
Why voice is different
You cannot barge in on a chatbot. Text turns arrive whole, so the entire behavior class above is absent from text training data. Voice also has no scrollback: in a chat window the user can reread anything, while on a call the shared state of the conversation lives only in the model’s head, including the awkward bookkeeping of what the caller actually heard versus what the model meant to say. Our example turns on exactly that distinction, five five versus five five seven nine. And because a backchannel costs the caller nothing, these situations are not rare corner cases; in the benchmark they occur every few turns, which matches what any customer service call looks like. In fact, in some cultures, such as in Japanese, backchanneling is expected. There, the absence of backchanneling usually means that the other party has stopped following the conversation.
You can tell a model how to converse. It may not listen.
Each conversation comes with a system prompt that spells out the workflow and how to handle callers. This is not a gotcha where the model must guess the desired behavior; the instructions are in its context the whole time. Following instructions about what to say is close to a solved problem, with IFEval scores in the high nineties for frontier models. Instructions about how to conduct a conversation are another matter. On filler interruptions, every GPT-family audio model continues the utterance correctly less than a third of the time (7% to 31%). The Gemini 2.5 family manages 62% to 68%. And the newer Gemini 3.x line regresses sharply, back down to 13% to 32%. Newer and smarter does not mean better at this. That pattern looks less like a capability ceiling and more like a training default, the same conclusion the proactivity experiments pointed to: the behavior is latent, but nothing in post-training selects for it.
Designing the benchmark
The dataset consists of 45 synthetic conversations across 10 enterprise domains, each driven by a state-machine workflow, with 428 interruptions injected mid-utterance. As in ProactBench, the grading rubric is written before any model answers, so the goalposts are planted before the kick. Every response is scored on two axes: task fulfillment, a head-to-head comparison against GPT-4o Audio (0.50 means parity), and recovery quality, a pass/fail check against the type-specific criteria above. The announcement post has the pipeline details, including the blinding that keeps the user simulator from leaking undelivered content into the conversation.
Across 27 model configurations, the two axes disagree about who is best. GPT Realtime 2 wins task fulfillment (0.73) but ranks mid-pack on recovery quality; Gemini 2.5 Flash with thinking is the recovery champion (0.70) at a much lower task fulfillment score. Getting the job done and conversing well are different skills (and good models should be good at both). The correlation structure says the same thing:
The four axes of Audio MultiChallenge correlate at 0.65 to 0.93 with each other; recovery quality is the least correlated axis in the whole matrix. It is a separate capability, just as Recovery was for ProactBench. Again, no existing leaderboard predicts it, thus the need for a new benchmark.
More numbers
Task fulfillment decays as conversations grow: 24 of 26 audio models have negative depth slopes (this is to be expected). Note that the open-weight models decay more than 3 times faster than the closed ones, likely due to the fact that they weren’t trained as good conversationalists at all. The only models that hold steady across twenty turns are the reasoning-enabled GPT Realtime variants. On modality, audio input never beats text: statistically equivalent for Gemini, and about 8 points worse for the open-weight models, so today the microphone is a tax you pay for convenience. The LLM judge behind all these numbers was checked against human annotators, who agree with it at least as well as the humans agreed with each other; reranking the models by human votes changes essentially nothing.
The models that fumble our bank call are not missing a capability. The instructions were in context, some models follow them, and the rest were never trained to. The fix is the unglamorous kind: logs of real interrupted conversations, and a benchmark that notices when you’ve lost the thread.
This is work led by Ahmad Salimi with Wentao Ma, Yuzhi Tang, Dongming Shen, and Mu Li at Boson AI.
Paper: arXiv:2606.19595 · Code: github.com/boson-ai/ihbench · Data: huggingface.co/datasets/bosonai/IHBench