Tomorrow I am giving one of the invited talks in the Applied Data Science track at KDD 2026 in Jeju, Korea. It’s a builder’s tour of some of the audio stack at Boson AI in three parts: how we train models that can hold a conversation, how we design benchmarks that tell us whether the next model is actually better, and how we decide which benchmarks are worth running at all.
There’ll be a show and tell of the running system: voice cloning and audio understanding first, then a live call in Voice Studio. First audio in under a second (750 ms median in production), a web search issued mid-conversation and folded into the next spoken turn, and a language switch mid-interview. TTS covers 100+ languages and STT covers 94, so the switch happens inside one conversation. There is also a demo of the same audio stack driving a live avatar. That combination has not shipped yet; it is coming soon.
The rest of the talk:
Building it. The pipeline from datacenter to serving. We run our own 1 MW facility in Canada with 520 H100s for training, 400 A100s for inference, and 30 PB of storage. The raw material is 100M+ hours of crawled audio; careful segmentation, open source ASR, and LLM cleanup distill it into 10M hours of clean transcribed English, roughly 20 human lifetimes of listening. I’ll cover Higgs Audio v2 which uses an open weights LLM backbone and an acoustic/semantic tokenizer, to cover both noisy telephony input and beautiful output (our v3 model works a bit differently but that’s the story of another post). In production we chain an understanding model with a smaller generation model, every stage streams, and a barge-in truncates output within 0.125 s.
Evaluating it. Humans are expensive and humans are random, so the real question is when you can trust LLMs to help you improve LLMs. Three benchmarks, one per capability we care about: EmergentTTS-Eval scores hard speech (emotions, paralinguistics, foreign words, formulas) with a judge that tracks human raters within 95%; ProactBench measures whether a model infers what the user wants before being asked; IHBench measures what a voice agent says after an interruption, not just whether it stops talking. Each one calibrates its LLM judge against human raters before we let it grade anything.
Picking good benchmarks. There are 170+ public tasks and nobody runs them all. Benchmark scores live in a low-dimensional subspace, so submodular selection finds a small subset that predicts the rest; 5 of MMLU’s 57 subjects reach \(R^2 \approx 0.91\). Run in reverse, the same machinery ranks a new benchmark by how much it adds to the incumbents. Greedy selection ranks ProactBench’s Recovery axis #2 of 9 and IHBench’s recovery quality #2 of 6, which is the quantitative way of saying that the previous section was worth the GPU time.
Regular readers will recognize the last two parts: ProactBench, IHBench, and benchmark selection each have their own post with the details. And yes, we are hiring, in Santa Clara and Toronto.
Slides: AI for Humans (PDF, 9 MB) · KDD 2026 Applied Data Science track, Jeju, August 11.