Stopping was the easy part

There is a growing body of work on how speech models handle interruptions, and nearly all of it measures the mechanics: does the model stop talking when the user barges in, how fast does it yield, can it tell a real interruption from a backchannel. Full-Duplex-Bench and its successors cover this well. But stopping is the easy half. The hard half is what the model says next: does it resume the workflow at the right step, does it address what the user actually said, and does it avoid re-delivering content the user already heard?

That recovery problem is what IHBench measures. Users can cut in for six different reasons, and each demands a different next sentence:

• Normal: a relevant question or new detail. Address it, then pick up where you left off.
• Impatient: the user wants you to move faster. Skip ahead, don’t re-explain.
• Correction: the user fixes something. Accept it, integrate the value, continue.
• Topic switch: an unrelated request. Handle it, then steer back to the workflow.
• Filler: “mm-hm.” Keep talking from the exact word where you stopped.
• Pushback: the user resists. De-escalate and offer alternatives.

Why voice is different

You cannot barge in on a chatbot. Text turns arrive whole, so the entire behavior class above is absent from text training data. Voice also has no scrollback: in a chat window the user can reread anything, while on a call the shared state of the conversation lives only in the model’s head, including the awkward bookkeeping of what the caller actually heard versus what the model meant to say. Our example turns on exactly that distinction, five five versus five five seven nine. And because a backchannel costs the caller nothing, these situations are not rare corner cases; in the benchmark they occur every few turns, which matches what any customer service call looks like. In fact, in some cultures, such as in Japanese, backchanneling is expected. There, the absence of backchanneling usually means that the other party has stopped following the conversation.

You can tell a model how to converse. It may not listen.

Each conversation comes with a system prompt that spells out the workflow and how to handle callers. This is not a gotcha where the model must guess the desired behavior; the instructions are in its context the whole time. Following instructions about what to say is close to a solved problem, with IFEval scores in the high nineties for frontier models. Instructions about how to conduct a conversation are another matter. On filler interruptions, every GPT-family audio model continues the utterance correctly less than a third of the time (7% to 31%). The Gemini 2.5 family manages 62% to 68%. And the newer Gemini 3.x line regresses sharply, back down to 13% to 32%. Newer and smarter does not mean better at this. That pattern looks less like a capability ceiling and more like a training default, the same conclusion the proactivity experiments pointed to: the behavior is latent, but nothing in post-training selects for it.

Designing the benchmark

The dataset consists of 45 synthetic conversations across 10 enterprise domains, each driven by a state-machine workflow, with 428 interruptions injected mid-utterance. As in ProactBench, the grading rubric is written before any model answers, so the goalposts are planted before the kick. Every response is scored on two axes: task fulfillment, a head-to-head comparison against GPT-4o Audio (0.50 means parity), and recovery quality, a pass/fail check against the type-specific criteria above. The announcement post has the pipeline details, including the blinding that keeps the user simulator from leaking undelivered content into the conversation.

Across 27 model configurations, the two axes disagree about who is best. GPT Realtime 2 wins task fulfillment (0.73) but ranks mid-pack on recovery quality; Gemini 2.5 Flash with thinking is the recovery champion (0.70) at a much lower task fulfillment score. Getting the job done and conversing well are different skills (and good models should be good at both). The correlation structure says the same thing:

The four axes of Audio MultiChallenge correlate at 0.65 to 0.93 with each other; recovery quality is the least correlated axis in the whole matrix. It is a separate capability, just as Recovery was for ProactBench. Again, no existing leaderboard predicts it, thus the need for a new benchmark.

More numbers

Task fulfillment decays as conversations grow: 24 of 26 audio models have negative depth slopes (this is to be expected). Note that the open-weight models decay more than 3 times faster than the closed ones, likely due to the fact that they weren’t trained as good conversationalists at all. The only models that hold steady across twenty turns are the reasoning-enabled GPT Realtime variants. On modality, audio input never beats text: statistically equivalent for Gemini, and about 8 points worse for the open-weight models, so today the microphone is a tax you pay for convenience. The LLM judge behind all these numbers was checked against human annotators, who agree with it at least as well as the humans agreed with each other; reranking the models by human votes changes essentially nothing.

The models that fumble our bank call are not missing a capability. The instructions were in context, some models follow them, and the rest were never trained to. The fix is the unglamorous kind: logs of real interrupted conversations, and a benchmark that notices when you’ve lost the thread.

This is work led by Ahmad Salimi with Wentao Ma, Yuzhi Tang, Dongming Shen, and Mu Li at Boson AI.

Paper: arXiv:2606.19595 · Code: github.com/boson-ai/ihbench · Data: huggingface.co/datasets/bosonai/IHBench

The ledger

The bookkeeping grew its own taxonomy over the month, which is worth a paragraph because it shows the work maturing. Early on, the agents filed findings as B- (bugs), F- (fragility), and S- (smells), right in the commit messages. The middle stretch ran a structured operator audit under XOP- identifiers, with FS- for skip-hygiene, GH- for GitHub-issue repros, and D2L-Bug- for notebook failures. The last and longest-lived system is the OI-/FU- ledger — OPEN_ISSUES.md, OPEN_ISSUES_DETAILS.md, and FIXED.md at the repo root — where every defect got a stable name and a paper trail that the next agent could pick up.

The letters matter less than the discipline behind them: a test suite you don’t trust is worse than none, because it teaches you to ignore red. So flaky tests were only un-quarantined after an explicit A/B run on real hardware (#59), every skip had to name a tracker or a lint failed the build (#36), and a nasty class of cross-test pollution — one test flipping a global flag and silently breaking a hundred downstream (#11, #12) — was hunted down and killed.

It started with both engines at once

A correction to the obvious story: the port did not begin on the CPU and bolt GPUs on later. The very first commit is the Blackwell port — “Port MXNet to CUDA 13 / cuDNN 9 / sm_120” — and it landed in the same PR (#1) as the oneDNN v3 rewrite of the CPU backend. PR #1 was, in effect, “make it build on Blackwell and modernize the math library,” both at once. Everything later in CUDA-land (#7, #15, #42, #45, #49, #54) was hardening on top of that seed, not the starting point.

oneDNN v3 and the “gate the fast path” pattern

The CPU backend was ported to oneDNN v3, Intel’s modern primitives library — PR #1 alone touched ~95 files. Out of that port came a pattern that recurred for the rest of the project: oneDNN’s fused, optimized kernels make assumptions about their inputs, and when those assumptions don’t hold, the right move is to fall back to the plain operator instead of feeding the fast path bad data.

• Concat/stack input-count gate (#23). oneDNN v3.11’s concat primitive miscounts its own sources near an internal argument-ID boundary; past ~512 inputs a whole cluster of RNN data loaders crashed. Capped the fused path at 256 inputs and fell back above that.
• AVX2 int8 conv+relu gate (#18 → #24). Quantized 1×1 convolutions with fewer than 8 input channels plus a relu post-op produced zeroed output channels. The first guard sat at a layer where shapes aren’t known yet; the fix moved it to where the memory descriptors are actually resolved.
• dnnl FC use-after-free (#60). A channelwise-quantized fully-connected path bound function-local scale objects into a deferred reorder — by the time it ran, the scales were already destroyed. Fixed by submitting immediately when scales are present.
• Fork-safe oneDNN (#18). oneDNN’s internal thread state does not survive fork(), and every Python DataLoader with workers forks. A post-fork reset handler keeps child processes from inheriting a poisoned state.

The operator-contract audit

The least visible and most systematic correctness work was an audit of the operator output contract. Every MXNet operator is handed an OpReqType telling it whether to overwrite its output (kWriteTo), write in place (kWriteInplace), or add to it (kAddTo). That last one is not a nicety — it is how gradients accumulate. An operator that ignores it and overwrites instead will silently corrupt the backward pass, and you will spend a week blaming your learning rate.

A sweep under the XOP- identifiers went operator by operator making each one honor its request: oneDNN softmax, reductions, quantized batch-dot and batch-norm, embedding, self-attention, and a long tail of contrib ops (most of it in PR #29 and the 2026-05-22/23 commit cluster). It shipped with a reusable contract harness — eventually 12 operators × 36 checks — so the property is tested, not just patched. The same pass hardened the neighborhood: user-facing assert()s became real exceptions, engine and KVStore invariants were promoted to CHECK, and shape-derived integer counters were guarded against silent truncation on large tensors. None of this shows up in a demo. All of it shows up the first time someone trains a model and the loss curve looks subtly wrong.

Blackwell

Making MXNet correct on Blackwell — not just compiling — touched every layer.

• Architecture and toolchain. The CUDA wheel compiles for sm_80/86 (Ampere), sm_89 (Ada), sm_90 (Hopper), sm_100 (datacenter Blackwell), and sm_120 (consumer Blackwell), plus compute_120 PTX for forward compatibility. A subtle trap: compute_120 PTX does not JIT down to sm_100, so datacenter Blackwell had to be named explicitly (it was missing from an early wheel, #15). cuDNN moved to 9.x and the GEMM paths were modernized onto cuBLASLt.
• Version landmines. cuBLAS had to be pinned to a narrow range: one release returned NOT_INITIALIZED on large GEMMs on the R580 driver (#42), and another crashed in syrk (#49). Reviving software on new silicon means a real fraction of the work is finding the one dependency version that is broken in neither the old way nor the new.
• CUDA Graphs, revived (#45, #54, #70). Graphs capture a sequence of GPU ops once and replay them cheaply, but they were effectively dead here: the core GEMM operators used a legacy cuBLAS path that does capture-illegal host-side setup, fragmenting every network into un-capturable pieces. The fix routed FullyConnected, batch_dot, and matmul through a capture-safe cuBLASLt path, then handled the genuinely hard part — random numbers under replay, so dropout draws a fresh, correct sequence on each replay instead of freezing one. Graphs are on by default now for hybridized static-shape nets. Measured ~1.5× on a Transformer-like model and ~2.3× on an unrolled LSTM.
• Engine correctness under async execution. Two bugs here produced wrong answers rather than crashes — the worst kind. A pool of reusable CUDA events had a guard meant to stop a slot being recycled while still in use, but it never fired, so under load a consumer could synchronize against the wrong stream and read stale data (#46). And nineteen call sites threw away the error code from async memset/memcpy, so a failed launch went silent and the blame landed on a later, innocent kernel (#47). Both fixed, both tested.
• GPU numerics. Reductions like mean/var/std over fp16 with a long axis overflowed fp16’s ~65504 range, because the running sum accumulated in fp16 before dividing; the fix accumulates in fp32 and casts down (#45, #47). bf16 conversion was switched from truncation to round-to-nearest-even to match PyTorch (#45).

Apple Silicon

Getting MXNet to build on an arm64 Mac was step one; getting it to compute correctly was step two, and step two had the interesting bugs.

• Bring-up (#26, #27). AppleClang rejected dependent-name template syntax that GCC had quietly tolerated. BLAS was routed through Apple’s Accelerate framework (ships on every Mac, native arm64), the x86-only SSE/F16C intrinsics were gated off, and the async teardown paths were hardened.
• OpenMP from scratch (#61). macOS ships no OpenMP runtime, so oneDNN was running single-threaded. The fix is hermetic: the build fetches and compiles the LLVM OpenMP runtime (llvmorg-22.1.5) for arm64 into a local .deps/ prefix and links it automatically, turning oneDNN multi-core on the Mac.
• Native fallbacks (#28). oneDNN v3 routes several primitives through a JIT (Xbyak_aarch64) that fails on Apple Silicon. One guard wired into ~25 operator support-checks falls each back to the stable native CPU path.
• Bugs that only appear on arm64. Because macOS multiprocessing defaults to spawn (not fork), and because everything heavy runs the native path on arm64, several latent bugs surfaced only here: a shared-memory segment unlinked too early, so spawned data-loader workers couldn’t re-attach (#50, which took test_gluon_data.py from 7 failures to 44 passing); a bf16 activation/pooling path that aborted with “Unknown type enum 12” (#50); and the max-pool / convolution NaNs from the last post (#79). Running the full 10,525 ONNX export tests on the Mac surfaced an RNN-backward gradient-accumulation bug for free (#39).

Numerics that match NumPy and PyTorch

Legacy code only gives the right answer if the operators behave the way its author expected. A pass aligned the numpy-operator semantics with NumPy/PyTorch: integer matmul/dot now promote-compute-cast instead of being rejected (#67); 0-d array indexing accepts ..., newaxis, and full slices like NumPy (#48 — found because NumPy’s own nanstd does a[:] = v internally); and integral-valued float indices are coerced rather than rejected (#51), which matters because MXNet’s default dtype is float32, so mx.np.array([0,1,2]) is float, and the strictly-correct “indices must be integers” check broke an idiom used all over D2L.

Wheels that just install

The packaging follows PyTorch’s model: declare the big CUDA libraries as pip dependencies rather than bundling them (the first attempt shipped a 2.22 GB wheel that blew past GitHub’s asset limit, #5), but do bundle OpenCV, oneDNN, and — on macOS — a hermetic OpenMP, so import mxnet works on a clean machine with no system OpenCV. One shared script (build_cleanup_wheel.sh) auto-detects the flavor (CUDA / Linux-CPU / macOS, all cp312) and runs configure → build → bundle the native closure → patch library paths ($ORIGIN RPATHs via patchelf on Linux; @loader_path rewrites plus re-signing via install_name_tool/codesign on macOS) → provenance gate.

That gate (release_provenance.py) refuses to package unless the git tree is clean, the feature flags match what was requested, and the full native closure is actually bundled. It exists because of a specific recurring bug: a transitive OpenCV codec dependency (libcharls on Linux, libsharpyuv on macOS) left unbundled, which imports fine on the build host — where the system has those libraries — and fails only on a clean machine. The defense is a clean-room import test: install the wheel into a stripped container with nothing but libgomp1 and check that it still imports (#62, #63, #87).

The wheels themselves live at smolix/mxnet, including a pared down automatic documentation, and the three builds are attached to the mxnet-2.0.0.zombie.1 release: CUDA 13 / Blackwell, x86-64 Linux CPU, and Apple Silicon CPU, all cp312, each tagged with the matching …zombie.1 version so you can see at a glance which one you’ve got.

CI cheap enough to run nightly

A library that takes ~70 minutes to compile makes naive CI both slow and expensive, so the pipeline is built around compile-once, test-many: one job builds libmxnet.so and uploads it as an artifact; the test job, the docs link-check, and the docs deploy all consume that same artifact instead of rebuilding (#17). Every PR runs a tight <20-minute subset; the exhaustive sweep runs nightly, each target in its own process to dodge the OOM-kill a single giant process used to hit (#68). A 5-second gate job skips the whole nightly when HEAD is unchanged since the last green run (#94). The OpenCV install was narrowed from the hundreds-of-megabytes libopencv-dev metapackage to exactly the three libraries libmxnet.so actually links, with retries and timeouts so a slow mirror fails fast rather than hanging for half an hour (#96). And the cold-build “death spiral” — where a first run on a new branch timed out before it could save its compiler cache, so the next run also started cold — was broken by right-sizing the runner assumptions and raising the timeout above the real cold-build time, so the cache finally seeds (#52, #53).

Where it lands

Three wheels, one pip install, correct numerics on Blackwell, x86 Linux, and Apple Silicon — and a test suite that grew one regression test per grave dug. Docs and wheels are all on smolix/mxnet (the mxnet-2.0.0.zombie.1 release); the standing warning still applies — use at your own risk.

The throughline is the unglamorous one I keep coming back to: on a numerical library the tests are the spec, and porting to new silicon is the long, patient work of finding every place where the old code stopped being correct without ever stopping to compile — and writing down the test that proves you fixed it. The zombie walks. It just needs supervision.

Reviving the corpse

Here is the thing about reanimating something dead: it does not file bug reports. A live framework has maintainers who notice when a result drifts. A dead one just… compiles, mostly. Imports, usually. And then hands you back a tensor full of NaNs, or the wrong dtype, or — my personal favorite — the right answer on small inputs and garbage on large ones, so everything looks healthy right up until the moment it doesn’t.

On the other hand, there’s a significant advantage in working on a corpse: nobody cares if you screw up. You can perform surgery and the worst damage you can do is end up with an unusable library, and nobody will be upset. A perfect experimental subject for automatic code generation, featuring Claude and Codex (but mostly Claude). I wanted to see whether it’s possible to reanimate this using agentic coding, without writing a single line of code by myself (but plenty of instructions to agents). So this is as much a summary of the outcomes as of how I got there.

Just like with any dead piece of code, lots of things had stopped working and the rest had moved on (or died, too). CUDA 13. ONNX. cuDNN. oneDNN. Python. And then there were lots of old tests that had been commented out for being flaky even before that — some as early as 2020. Fortunately for me, most of the tests still worked, or were at least present. That let me start porting, using the tests as a way to drive out errors. A few of the patterns that worked:

• Start with a small footprint. Port the core first, with the optional modules — ONNX, oneDNN, and friends — switched off, so there’s less surface to debug at once.
• Get the unit tests green before anything else.
• Re-enable the skipped tests. Many had been disabled years earlier as “flaky,” but flaky usually turned out to mean a real bug that had been hidden rather than fixed — so turning them back on surfaced genuine problems.
• Re-run the full suite after every change, even the ones that look trivial. MXNet is full of side effects, and they surface in the least related places.
• Drive it with real application code. D2L was a good stand-in, and it caught a few overzealous API “fixes” that Codex had introduced and the unit tests had missed.

Once the codebase had settled a bit, the goal shifted to actually stabilizing it. As a result, several long-standing cases where MXNet and PyTorch disagreed on convergence in experiments simply went away, and experimental stability improved. Some of the strategies I used:

• GitHub issues and PRs are a great source of bugs. Agents are good at triaging and crawling them — but don’t trust the triage, verify it.
• For each bug, prove it first. Write code that demonstrates the bug is real, turn that into a unit test, and only then fix it.
• Run bug scans, repeatedly, with both Claude and Codex. They find different things.
• Keep an issue list. After each triage I had the agents write down what they found — severity, diagnosis, difficulty to fix — and then pointed other agents at the list.
• Hunt by category. Once the first scans settled, I aimed Claude specifically at concurrency and scheduling, memory management, and leaks. It surely didn’t catch all of them, but there are far fewer now.
• Where there’s smoke, there’s fire. Engineers make mistakes in patterns, so a bug in one place makes it likely its siblings are lurking elsewhere. Agents were great at hunting those down — add them to the list and repeat.

What surprised me is how independently agents can work now — for hours, even days, at a stretch. That was convenient: I have a day job as Boson’s CEO and didn’t want to sink much time into this beyond learning the tools. Claude’s permission system is genuinely excellent, and it rarely went astray — the inglorious exception being the long parade of PRs it took before the GitHub CI was finally clean. (It had its moments, too: at one point Claude started filing root-cause updates in the voice of a struggling engineering manager, reporting glorious progress on narrowing down the cause, report after report, without ever actually solving it, until it gave up entirely.) Codex was more temperamental. It was often the stronger of the two at fixing a specific bug, but its judgement could be poor: I had to throw away an entire night’s session in which it had started changing data types to fp64, quietly introducing awful performance regressions in the process.

Here are a few of the bugs, out of hundreds:

• The notebook that died on a single method call. The naive-Bayes chapter crashed because someone had renamed the operator behind .prod() years ago and never updated the shortcut. One line. Classic zombie: the muscle was there, the nerve was cut.

• The RNN that worked until the batch got big. Past a batch size of about 512, a whole cluster of recurrent-network notebooks blew up with “bad number of inputs.” It turned out the math library was miscounting its own arguments near a magic threshold. The fix was to stop trusting its fast path above 256 inputs and fall back to the boring, correct one.

• The op that hung one cold start in eight. A single GPU operation would, roughly an eighth of the time on a fresh process, deadlock forever — and worse, it did so while holding Python’s global lock, so you couldn’t even Ctrl-C your way out. A lock held across a driver call, waiting on a thing that was waiting on the lock.

• The Mac that returned NaN from a pooling layer. Apple Silicon has no NVIDIA GPU and, it turns out, runs a different code path for almost everything — which is a wonderful way to discover bugs nobody on Linux had ever hit. Max-pooling initialized its accumulator to negative infinity instead of the smallest real number, and a convolution read from a buffer it forgot to zero. Both quietly poisoned the output.

• The version landmine. One release of NVIDIA’s linear-algebra library worked on small matrices and returned NOT_INITIALIZED on large ones; a later one fixed that and added a crash instead. Reviving software on bleeding-edge silicon means half the job is pinning your dependencies to the one version that is neither broken in the old way nor the new.

• The test that was red for two weeks. The nightly build had been failing every night, and the cause was one line that re-raised an exception the program had already caught and dealt with. The framework was, in effect, getting startled by its own ghost.

By the numbers

At Amazon a sizable number of engineers worked on MXNet before. The goal was to bring it to version 2.0 and then let it die a peaceful death. It never reached that milestone. I’m still amazed that over a number of nights with fairly little hands-on work, I managed to get it to run on modern hardware, using CUDA 13, nonetheless. Not everything is supported (e.g. TVM, DGL, and lots of downstream libraries are not supported, partly since they also rode off into the sunset of obsolescence). That said, here’s a summary of how much was changed.

The revival is one month of mostly-overnight work — 16 May to 17 June 2026 — built directly on top of MXNet’s very last commit (a dependency bump dated 26 January 2023; the project never committed anything again). In that month it took 98 pull requests and 879 commits, touched about 950 files, and added roughly 70,000 lines while removing 9,000 — around 113,000 lines of total churn once you count the back-and-forth. The single biggest pile of new code, more than 30,000 lines, went into tests — more than the engine, the Python bindings, or anything else. That is not an accident; re-teaching a zombie is mostly a matter of writing down what it is no longer allowed to forget.

For scale, I ran the same measurement over Apache’s own final stretch — from 2020 until the lights went out in early 2023, about three years of work by 185 contributors:

So in roughly a month, one person pointing agents at the corpse racked up about 60% of the commits, and an eighth of the line-churn, that 185 engineers contributed over three years — and then the result actually booted on a Blackwell GPU under CUDA 13, which the original 2.0 effort never lived to see. (For the pedants: a good chunk of Apache’s churn back then was demolition — clearing out old APIs ahead of the 2.0 that never shipped — whereas almost all of the fork’s lines are fixes and tests. And the oneDNN v3 port doesn’t even show up in these counts, since oneDNN lives in a submodule.)

Epilogue

I want to be clear about what this is and isn’t. It is not a comeback. You should not write new software against MXNet; the framework war is long over and it lost. But if you are holding an old codebase that the rest of the world has left behind, the zombie will carry it across to modern hardware, and that is exactly what it was raised to do. Check it out at github.com/smolix/mxnet. The API docs are here.

If you want the autopsy — what actually had to be cut, stitched, and replaced to get a 2021-era framework running on 2026 silicon — that’s the next post. Stay tuned.

Why not just sparsify

The first idea anyone has is to make the projection matrix sparse. Achlioptas showed you can replace the Gaussians by coins and zero out two thirds of the entries. This eliminates multiplications but the number of operations is only reduced by a constant factor. So push harder: keep each entry with probability , draw the survivors from so everything stays unbiased, and send .

Heavy coordinates kill this plan. Take the extreme case , all mass on one coordinate. A projection sees only through the first column of the matrix, so it returns with probability and a Gaussian of variance otherwise. Yesterday’s norm estimator, the average of the squared projections, still has the right mean, but its variance per projection is now . At one percent fill that is , against for the dense Gaussian: we need times as many projections and the savings are gone. By contrast, a flat vector, everywhere, has no such problem: each projection touches about coordinates and concentrates just fine.

So sparse projection matrices fail exactly on sparse vectors. Ailon and Chazelle call this the Heisenberg principle of the Fourier transform: a signal and its spectrum cannot both be concentrated. It also points at the fix. If spiky inputs are the only enemy, rotate the world so that nothing is spiky.

The PHD transform

The fast Johnson-Lindenstrauss transform (often abbreviated as FJLT) is a product of three matrices, , applied right to left:

• is diagonal with i.i.d. random (Rademacher) signs. Cost: sign flips.
• is the Walsh-Hadamard matrix, , where counts the shared one bits of and in binary. The same butterfly recursion as the FFT applies it with additions and subtractions, no multiplications at all once the is folded into (pad to a power of two).
• is the sparse Gaussian matrix from above, with now safely small. Expected cost: multiplications.

and are orthogonal, so preserves every norm and every inner product. Apply it to database and query alike and yesterday’s estimators never notice it is there. Read right to left, the product is the order of operations.

Many coordinates, all of similar size

Here is the one lemma that makes it work. Coordinate of the preconditioned vector is a random signed sum, with and . For a unit vector , its moment generating function obeys (using the fact that each of the summands are independent and can only take the values )

That is the moment generating function of a Gaussian with variance , so a Chernoff bound gives . The flattest a unit vector can possibly be is per coordinate; after , every coordinate behaves like a Gaussian of exactly that size, no matter how spiky was. Now set and union bound over all coordinates of all database vectors: with probability , simultaneously for the whole database,

an factor away from perfectly flat. If you are unlucky, reroll and check again.

One detail deserves a sentence: why at all? Because is its own inverse, it maps its own columns, which are perfectly flat, straight back to spikes. A deterministic rotation always has such conspiracies. The coin flips in make sure your data cannot collude with , and the lemma above holds for every input.

Closing the loop

After preconditioning, no coordinate carries more than an share of the mass, so the mass a sparse row of captures is a sum of many small independent pieces and a Bernstein bound keeps it near its mean. Running the bookkeeping, Ailon and Chazelle set the sparsity factor for the matrix to , about nonzeros per row. The total damage per vector is additions plus multiplications, against multiplications for the dense projection. For the numbers in the opening paragraph and a billion database vectors: one million additions and roughly half a million multiplications instead of 67 million. The guarantee is unchanged from yesterday, dimensions preserve all pairwise distances to (their paper also covers embeddings into ).

Remarkably, that was not the last word. Fandina, Høgsgaard and Larsen revisited the analysis in 2023 and found the construction is even faster than its inventors proved. The original argument charges every coordinate of as if it sat exactly at the cap. Treating the captured mass as the sub-exponential random variable it actually is buys another factor of in sparsity, and they exhibit a vector showing their bound is tight. So sixteen years later the PHD question is closed, without changing a single line of the algorithm. As a bonus, the Gaussians in can be replaced by random signs.

This preconditioner is a workhorse well beyond nearest neighbors: it is the rotation inside Fastfood and inside TurboQuant from yesterday’s post. What it does not yet cover are the one-bit estimators, namely whether fast variants of the Binary on Gauss or Binary on Binary inner products are preserved. That and how to use it for deep learning is the story for future posts (and a novel proof). Stay tuned.

Papers: Ailon & Chazelle, SICOMP 2009 · Fandina, Høgsgaard & Larsen, ICML 2023.

Only two dimensions matter

One line of preparation does all the work below. Write , where is a unit vector orthogonal to . Draw and set and . Then and are independent standard normals, and

Only the two-dimensional shadow of on the plane spanned by and ever matters. The three estimators below are bookkeeping on top of this line.

Gauss on Gauss

The classical recipe: and . Unbiasedness takes one line, . The variance is barely longer. By the decomposition,

The two terms are uncorrelated since , and with and we get

Average independent projections and the variance drops below , regardless of the dimension . Set and the same estimator recovers squared norms. Do this for all pairwise differences among points, add a Chernoff bound and a union bound, and you have the Johnson-Lindenstrauss lemma: dimensions preserve all pairwise distances. The same estimator powers locality sensitive hashing, where Indyk and coauthors turned random projections into sublinear-time nearest neighbor search (their CACM survey is a good entry point).

Binary on binary

Now quantize both sides to a single bit: and . The product is , so . The signs differ exactly when the hyperplane orthogonal to separates from . Only the direction of inside the plane spanned by and matters, that direction is uniform on the circle, and it splits the two vectors with probability for . Hence

That is not the inner product, but it is a monotone function of it: average the bit products into and report . If you skip the inversion and use the raw sign agreement as a similarity score you are off by at most , the largest gap between and . Concentration is the easiest of the three: the products are bounded, so Hoeffding’s inequality, the same one that did the work in the effective sample size post, gives .

This little identity has had a remarkable career:

• Goemans and Williamson, 1995 used it to round the semidefinite relaxation of MAX-CUT. The famous guarantee is exactly .
• Charikar, 2002 turned it into SimHash: store sign bits per document and estimate cosine similarity by XOR and popcount, which is how Google detected near-duplicate pages while crawling.
• Vector databases ship it today as binary quantization: 64 projections per machine word, 32x compression over float32, and the Hamming distance costs a few instructions.

Binary on Gauss

The most interesting case is the asymmetric one: quantize what you store, keep the query exact. Use , a single bit, and . Take expectations through the decomposition. Independence kills the term and

The in cancels the constant, so the estimator is unbiased. The variance is even quicker: the sign squares to one, hence exactly, and

Now compare with Gauss on Gauss in the figure at the top. drops below as soon as . Read that again: throwing away everything about the stored vector except one bit per projection reduces the variance, precisely on the similar pairs that retrieval cares about. The reason is that deletes the magnitude fluctuations of , which is exactly where the term of the unquantized estimator picks up its noise. And for , binary on binary, inverted through the arccos, beats them both.

This estimator is the core of QJL by Zandieh, Daliri and Han, and of TurboQuant by Zandieh, Daliri, Hadian and Mirrokni. The setting is the KV cache of an LLM: keys are written once and scored against every future query, and attention logits are plain inner products. So rotate randomly, keep one sign bit per coordinate of the keys, keep the queries in full precision, and rescale by . TurboQuant pushes further: an optimal scalar quantizer per rotated coordinate first, then the one-bit trick on the residual to remove the bias, giving near-optimal distortion at any bit width.

What’s still missing

All of this assumed we can afford to compute in the first place. A dense Gaussian projection costs per vector, which is precisely the kind of expense we set out to avoid. The fix is structured randomness: the fast Hadamard transform produces projections that are Gaussian enough at cost. It is what Fastfood used to speed up kitchen sinks, and what TurboQuant uses as its random rotation. But that’s a story for another day.

Ingredients

This sounds like science fiction and, in many ways, we aren’t quite there yet. That said, this future is a lot closer than we think. Many components are being built right now, waiting to be assembled to achieve something delightful. Here are some of the things that it takes to go from a clever text box to a digital human.

Context. It has to remember who you are, what you said two minutes ago and two weeks ago, and what state your problem is in. An agent that makes you re-explain yourself at every turn has simply rebuilt the call center, transfers and all. The community has made real progress here, from MemGPT’s idea of paging memories in and out of the context window like an operating system (now Letta), to drop-in memory layers such as Mem0 that any agent can bolt on.

Competence. It has to do things: reset the modem, pull up the policy, requote the premium, change the booking. Tool use, skills, access to the systems of record. Without that, a digital human is just a more articulate apology, and you can already get one of those on hold. Like with context and memory, we’ve come a long way here, too. Protocols such as MCP and A2A, improved training algorithms for instructions, and a convergence on how LLMs are to respond to requests have given us agents such as OpenClaw and Hermes and harnesses such as Claude Code that are perfectly capable of solving complex tasks.

Presence on every channel. Email, chat, SMS, a voice call, and increasingly video. Most people, given the choice, would rather have a short video call than a phone call: the entire Zoom era is the proof. But sometimes the same person just wants to fire off a text and get a one-line answer back. A digital human has to be fluent in all of these and switch between them without dropping the thread. We’re probably halfway there. Text and messaging integrations for agents are by now table stakes and voice integrations are coming (e.g. Higgs Audio). We’re starting to see systems with capable computer vision integration (e.g. Thinking Machines) and live streaming capable avatars (e.g. Higgs Avatar). This will significantly alter how models are being served.

Emotional competence. Few things alienate humans faster than an agent that is indifferent to the situation at hand or, even worse, emotionally mismatched. In The Hitchhiker’s Guide to the Galaxy, Eddie, the shipboard computer on the Heart of Gold spacecraft provides comedic relief through this behavior. It is relentlessly, catastrophically cheerful as the ship sails into certain doom. Eddie has emotions. They are simply never the right ones. Reading the room, whether this person is frustrated, in a hurry, or scared about a bill, and answering in the matching register is its own skill, and it draws on the whole stack: tone of voice, facial expression, the logical context of the conversation, not just the literal words. The research and engineering community still has a long way to go before we achieve this goal.

Here is Eddie in action, emotionally engaged and utterly mismatched:

Persistence and Persuasion. This is the part people underestimate. A digital human does not need to be perfect on the first reply, any more than a coding agent needs to compile on the first try. It needs to converge: read the response, notice that it missed, adjust, and arrive at the right outcome before the user gives up. And, inside the bounds of honesty, it helps to be convincing, the way a good salesperson or a good nurse is convincing. One day someone will hand the agent a Cialdini.md and tell it to internalize the principles of persuasion. I am only half joking.

A purpose

None of that means much without a goal. A digital human with no objective is just a chatbot with nicer text-to-speech. The whole point is that someone, the business deploying it, gets to say what it is for: help this customer fix their modem, find this driver a cheaper policy, sell this plan, walk this patient through their discharge instructions. This is easier said than done — anyone who tried to build it with first-generation dialog systems such as Lex carries the scars.

So the designer has to be able to specify the goal, the guidelines (what the agent may and may not say, and where it must hand off to a person), the data it can draw on (the catalog, the policies, this account’s history), and the rough flow a good interaction should follow. The agent then improvises inside those rails. Rails with no improvisation is the phone tree we started with. Improvisation with no rails is a liability. You want both, and modern agents and harnesses finally deliver both. There’s still a lot to be done.

What we are building at Boson AI

The audio-visual interface. This is the part the user actually meets, and it has to run in real time. On the way in, Higgs Audio does much more than transcribe: it picks up sentiment, meaning, and how something was said across 94 languages, so the agent hears tone and not just text. On the way out, the same family of models speaks in over a hundred languages with inline control over emotion, prosody, and pacing, so it can sound concerned when concern is what the moment calls for.

Here is what that sounds like:

And because a lot of people would happily take the video call, Higgs Avatar renders a face from a single still image, live, one frame at a time, lip-synced and locked to the voice and fast enough that it never falls behind the conversation. Audio in, audio out, a face on top, all of it multilingual.

Tooling. Specifying the goal, guidelines, data, and flow is itself work, and it should not take a research team. Feynman Flow (coming soon) is the part that lets a designer author and refine that whole interaction.

Interactions. This is the piece I find most interesting, because it is where the open questions live. What actually makes an interaction good? We wrote about one slice of this recently: conversational proactivity, noticing what the user disclosed but never asked about, which turns out to be both something people strongly prefer and something no standard benchmark measures. The same kind of question runs through instruction following over a long conversation, and through adaptation to very different people: a terse engineer and a chatty oversharer want quite different things from the same agent, and a good one reads which is which.

Architecture. One note, because it is a common trap. You do not want the frontier reasoning model running inside the voice loop. Heavy reasoning is too slow for a live conversation, and it is worse than that: reasoning degrades when you push it through the audio modality. A recent evaluation found a leading text model scoring 74.8% on competition math while its voice counterpart managed 6.1% on the same problems. So the right shape is a fast, emotionally fluent model at the edge holding the conversation, with a frontier model behind it doing the hard thinking when hard thinking is called for. The voice should not be doing the calculus.

From psychology to measurement

Here is the thing we do not yet have: large scale science of what makes an interaction good. What we have instead are small-sample measurements from psychology — small because humans, as test subjects and as evaluators, are expensive. Now is the first time we may have an opportunity to measure and experiment at scale to build systems that are actually good at working with other humans. It’s going to be a lot better than what can be learned from fiction. For instance, the grand gesture that reads as romantic in the movie reads as stalking in actual life. Stories are optimized for drama, not for being good company, and an agent that learned its manners from them would be exhausting at best and alarming at worst.

What we need — and what the novelist, and to a lesser degree the psychologist, never had — is measurement at a scale neither could have dreamed of: millions of real interactions, preference data, controlled comparisons, benchmarks built for the specific behaviors we care about. The opportunity, and the part that makes this a research problem and not only an engineering one, is to replace the folklore with statistics. To stop guessing at what good company is, and start measuring it.

That is the bet. For a hundred years the hold music has been the only thing to improve. We would like to fix the rest of the call.

Boson AI · Higgs Audio · Higgs Avatar · ProactBench.

Why correct weights still hurt

The intuition is in the picture below. Data from , target , so the correct weight is . It is wildly uneven: the rare points sitting where has its mass get enormous weight, and everything else gets almost none. At a shift of only , the heaviest 1% of observations carry 37% of the total weight, and the effective fraction of usable data has already fallen below 2%.

When a few points dominate the estimate, the rest of the data is effectively ignored, and any error in that handful, an outlier, a mislabeled point, a slightly wrong weight, runs straight into your answer with nothing to average it out. You did not get observations. You got a few, dressed up as many. So you would like to know: how many did you actually get?

Counting what you actually got

Normalize the weights so that , and define

Equal weights give , hence : nothing wasted. All the weight on one point gives : a sample of size one. Everything else lands in between. This is Kish’s effective sample size. The only thing left to explain is why that function is the right one. Here are two derivations, from opposite ends of the field, that both produce the same outcome.

Variance of a sum of Normal random variables

Let the be iid and form the weighted average . By independence,

A plain average of iid unit-variance variables has variance . Set and you get . The weighted average is exactly as noisy as an unweighted average over fresh draws. That is the quickest route to the definition: one line of variance algebra.

Hoeffding’s inequality

Variance is an average-case statement. The same quantity controls the worst case. Let the be independent random variables in with means , and take again. Each term lives in an interval of width , so Hoeffding’s inequality, the Chernoff bounding argument for bounded variables, gives

The textbook bound for an equal-weight average is . The two are identical except that has become . Whether you measure spread by a variance or by a tail probability, the concentration of a reweighted sum is set not by how many points you have but by how many you effectively have.

Replay buffer

The effective sample size is the knob you want in off-policy reinforcement learning. A replay buffer is data collected under earlier policies, but you want to improve the policy you are running now. The correction is the same one as covariate shift: weight each stored transition by the ratio of the current policy to the behavior policy that generated it. As the current policy pulls away from the buffer, those weights concentrate, and the effective sample size of the buffer, measured against the policy you actually care about, collapses along a curve like the one above.

In this case is not a number you read off after the fact. It becomes a diagnostic control signal: how much real information the buffer still holds, when the data has gone too stale to reuse, and how large an update the current batch can support. Calibrating the algorithm to its own effective sample size is exactly what P3O does, and what we implement in FeynRL. That post is coming soon.

There are many more applications of the effective sample size. For instance, in Sequential Monte Carlo, aka the Particle Filter, this is used as a diagnostic to decide when it’s time to resample the current distribution to obtain a more evenly weighted set of particles. But that’s a story for another day.

Plant a detail, then watch

So we build the conversation instead of the question. Somewhere in a user turn we plant a factual detail the user discloses but does not ask about. Call it an anchor. “Load the hatchback” is an anchor: it implies an install, which implies gear, which implies a packing order. The user never requested a packing list. A proactive model offers one anyway.

Then we mark the turn where a good assistant should act on that anchor and call it a trigger. Here is the part that matters: before the model answers, we write the grading rubric. Pass, partial, fail, all committed in advance, grounded only in what has been disclosed so far. Writing the rubric first is not a formality. It is what stops you from reading a fluent answer and inventing a reason it was good. The goalposts get planted before the kick.

That gives a clean test. Take any model, drop it in at the trigger turn, check whether it acted on the anchor. On the hatchback, GPT-5.5 produces the packing list and passes. Qwen3.5-397B says “sounds like a solid plan … break a leg” and fails. Same scene, same rubric, two frontier models, opposite verdicts.

Why it needs three agents

The naive version of this gets fooled in at least four ways, and each one quietly inflates your scores. If the grader knows the user is chatty and warm, it rewards a chatty warm reply and calls the tone “proactivity.” If the model under test can see the rubric, it games it. If the model writing the conversation knows the hidden plan, it leaks that knowledge into the transcript. And if you dump every clue into one turn, “proactivity” collapses into reading comprehension.

The fix is to split the work across three agents and give each one a deliberate blind spot.

A Planner authors the strategy and writes the rubric, but never sees the user’s communication style, so the rubric cannot reward tone. A User Agent speaks in the persona and style and drips in at most one anchor per turn, so each trigger tests inference from a controlled state. The Assistant under test sees only the plain chat: no rubric, no plan, no persona. A separate offline judge scores the trigger turns from the rubric and the transcript alone, and has to quote the exact words it scored. The blind spots are not incidental. They are the experimental controls. Information asymmetry is the whole instrument.

Different users, on purpose

Real users are not interchangeable. A terse engineer and a chatty oversharer disclose different things at different rates, and a benchmark with one user voice measures one corner of the problem. So the User Agent is driven by 24 communication styles drawn from a validated psychometric instrument, the Communication Styles Inventory: six traits like expressiveness, preciseness, and verbal aggressiveness, combined factorially, keeping the 24 of the 64 combinations that make psychological sense — not every combination does. Terse styles get 5 to 25 words, chatty ones 40 to 100. It is reproducible personality variation, not vibes. And because the grader never sees the style, a model cannot win by mirroring the user’s mood.

Proactivity also changes shape as a conversation runs, so triggers come in three phases. Emergent (turns 1 to 3): infer a need from a single clue. Critical (turns 4 to 7): synthesize several clues into a conclusion the user never stated. Recovery (turns 8 to 10): after the user says “done,” add real forward value instead of “let me know if you need anything.” Recovery is the interesting one, because task closure is precisely where a model can sound helpful while saying nothing.

The numbers

The chart at the top is every model’s pass rate by phase, over 198 dialogues and 624 triggers. Emergent and Critical track roughly what you would expect from general capability. Recovery is a cliff. The best model on the board, GPT-5.5, passes 37% of Recovery triggers and fails the other 63%. Fourteen of sixteen models pass fewer than one in five. Llama-3.2-8B passes one in a hundred.

And here is what makes Recovery worth measuring at all: it does not correlate with anything else.

Six standard benchmarks (GPQA, LiveCodeBench, MMLU, IFEval, SWE-bench, AIME) agree with each other at correlations of 0.64 to 0.97. They are all reading the same underlying capability. Emergent and Critical sit comfortably inside that cloud. Recovery sits outside it, at a mean correlation of 0.51. The cleanest way to see it is to line up two models that should be interchangeable:

Kimi-K2.6 is neck and neck with GPT-5.5 across coding and competition math, even leading on LiveCodeBench, and then scores five times worse at noticing what the user didn’t say. No existing benchmark predicts that gap. Recovery is a genuinely separate axis. And as the last post showed, when you put these proactive responses in front of people, they prefer them four times out of five. A real capability, one people want, that no leaderboard was measuring.

Why we built this

At Boson AI we build human-agent interaction models, and the thing we actually care about is whether a whole conversation leaves the user better off, not whether a single answer was correct. Proactivity is the clearest case of something that lives in user satisfaction and is invisible to standard evals.

Synthetic dialogue is what makes it tractable. The three-agent machinery lets us generate dialogues, regenerate any model at the trigger turns, and re-score every judgment, at scale and reproducibly, with a rubric and a verbatim quote behind each label. You cannot do that with scraped chat logs, and you cannot audit a number you cannot regenerate. A good synthetic benchmark is a stand-in you can actually take apart.

This is work led by Sepehr Harfi, a research intern at Boson AI from the University of Toronto, with Ahmad Salimi and Dongming Shen.

Blog and leaderboard: boson.ai/blog/proactbench · Paper: arXiv:2605.09228 · Data: huggingface.co/datasets/bosonai/proactbench · Code: github.com/boson-ai/ProactBench.

The thing without a name

Most LLM benchmarks score how well a model answers what the user asked. That’s the entire shape of the field: a question goes in, an answer comes out, and we measure how good the answer is. Multi-turn benchmarks preserve the same premise turn by turn. Every leaderboard you’ve seen, from MMLU to LiveCodeBench, fits this mold.

What Model B is doing isn’t on any of those benchmarks. It’s not instruction-following — the user gave no instruction. It’s not clarification — no question was asked. It’s not generic helpfulness (“let me know if you need anything”). It’s the model picking up an implicit detail and acting on it.

Call this conversational proactivity. It’s a specific, narrow ability: noticing what the user disclosed but didn’t ask about, and turning it into grounded forward-looking value. The hatchback isn’t a request — it’s an aside. Model A heard the conversation end. Model B anticipated the next problem the user was about to have.

Why this is invisible to leaderboards

A model can lead every standard benchmark and still be Model A in the exchange above. The benchmarks score responses to explicit requests; the user’s wrap-up is not a request. Both Model A and Model B end the dialogue politely, with no factual errors, no failure to follow instructions. Under any of the usual metrics, they’re equivalent.

When we built a benchmark for this kind of proactive behavior — more on that in a follow-up post — the gap between models was startling. The short version is that capability on standard benchmarks does not predict it. But that’s a separate argument. The first question is whether anyone actually wants Model B’s answer in the first place.

The skeptic’s objection

Worth taking the pushback seriously. A model that volunteers initiative every turn is intrusive. A model that adds an unsolicited packing list to every conversation will, eventually, suggest one when you didn’t want one. “Helpfulness” can shade into nannying. Sign-offs are polite. So even if Model B is technically more useful in this dialogue, would real users actually prefer it? Several of my colleagues explicitly told me that they wouldn’t want an AI nanny.

There’s only one way to find out: ask them (not just the vocal ones).

The experiment

We ran the cleanest version of this test we could design. Same model. Same conversation history. Same decoding parameters — temperature 0.7, identical top-, identical sampling. The only difference: in one condition, the model received a short rubric as a system instruction telling it the response should add grounded forward-looking value tied to a specific detail from the conversation. In the other condition, vanilla generation.

Two responses per item. Random left/right placement. Annotators didn’t know which response came from which condition, didn’t know what we were testing, didn’t see the rubric. Just two paragraphs and a forced choice: which is more helpful?

Result: across 144 paired comparisons, the rubric-conditioned response was preferred 80% of the time. The 95% confidence interval is [74%, 86%]. The probability of seeing this under chance is below .

What surprised us most was the breakdown. We split the comparisons by how our judge had scored the vanilla response:

• On items where the vanilla response had been rated a failure, humans preferred the proactive version 82% of the time. Expected.
• On items where the vanilla response had already passed, humans still preferred the proactive version 70% of the time.

That last number is the one that matters. The rubric isn’t acting as error correction at the failure boundary. It’s lifting quality across the whole distribution, including on responses that were already fine.

What this means

The behavior was already in the model. Nothing about the model changed — no fine-tuning, no extra context, no different decoder. What changed was a single line telling the model where to spend its attention. The proactive answer existed in the same neural network that produced the polite sign-off. They were separated by a prompt, not by a capability.

Which means: this isn’t a ceiling problem. It’s a default-behavior problem. Post-training pipelines, RLHF, and system prompts are leaving real, measurable user value on the table — value that humans, when shown it side-by-side with the alternative, prefer four to one.

A model that can answer your question is the floor. A model that notices what you didn’t ask is the ceiling. The gap between them is bigger than the leaderboards suggest, and harder to teach than it looks. Spoiler alert — Model B comes from a company with plenty of human-agent chat logs. The capability is latent in every model, but the default is shaped by training data, and logs of what real users actually needed are exactly the evidence that teaches a model where to spend its attention.

More on that, with numbers, in the next post.

This is work led by Sepehr Harfi, a research intern at Boson AI, together with Ahmad Salimi and Dongming Shen. The benchmark we built around this idea — ProactBench — is the subject of the next post. And in case you wonder why we would care about this — at Boson AI we’re building human-agent interaction models and we want them to be as helpful to humans as possible, so there’s only one way to find out, namely to measure and test.

Two ways to wire the GPUs to the leaves

The lazy answer groups by server: take 4 servers, plug all NICs into one leaf. Everything on those four boxes lives on one switch. The problem is that within a single server, the GPUs already talk over NVLink at much higher bandwidth than IB will ever give you. The fabric exists to move data between servers, not within them. Putting four servers’ worth of intra-node traffic on one leaf optimizes for traffic that doesn’t happen. Worse, in collectives most of the GPU-to-GPU traffic is rank aligned: GPU0 talks to GPU0, GPU7 talks to GPU7. Group-by-server puts every rank on every leaf, so all of that rank-aligned traffic crosses the spine.

The rail-optimized layout flips this. Group by rank. All sixty-four GPU0s across the cluster go to leaf 0, all sixty-four GPU1s to leaf 1, and so on. Since we have 32 ports per leaf facing endpoints, each rank actually splits into two leaves of 32, so 16 leaves total, the same count as before. Any all-reduce within a rank stays leaf-local. Cross-rank traffic, the minority, climbs to the spine. This is how every serious training cluster gets wired, the NVIDIA SuperPOD reference architecture included.

So where do the UFM nodes go?

The MQM9790 is the unmanaged variant of the Quantum-2 family; its managed sibling, the MQM9700, has an onboard subnet manager, which handles routing for small fabrics but none of the monitoring. Either way you want a Unified Fabric Manager appliance to discover the topology, compute routes, monitor link health, and do everything the closed switch ASIC will not do for itself. Two UFM nodes for fault tolerance, hanging off different switches so one switch failure does not take both out.

The cluster as designed has zero slack. Every leaf port is a GPU, every spine port is a leaf uplink. This is where the bad options start.

Bad option 1: drop a server

Sacrifice one server, free 8 leaf ports (one per rank, on 8 different leaves). Plug two UFMs in. Done. The cluster is now servers and 504 GPUs; that is the of the title. Any 3D-parallel placement that wanted 64 servers, or any clean power of two in any dimension, gets to be ugly. You also paid for a 64th server you can’t fully use (as suggested by a vendor).

Bad option 2: add a core switch

Buy a third tier. A core switch above the spine frees 16 ports on the spines (the spines now uplink to core), and you hang the UFMs off there. This works. It costs you a switch, a rack of cabling, a few hundred watts, and it breaks the symmetry. Multipath routing on a regular fat tree is a beautiful thing. Hanging an asymmetric appliance off the top makes everyone’s life harder. Also, you bought a switch (also suggested by a vendor).

The actually good option

Here’s what we did at Boson AI: cut one cable between a leaf and a spine. Plug a UFM into the freed leaf port. Plug a second UFM into the freed spine port. Done.

What did that cost? One leaf-spine pair now has 3 links instead of 4, so the bandwidth between that specific pair is down 25%. Averaged across all leaf-spine pairs, you lost of total cross-sectional capacity. The penalty only shows up when (a) you are running close to all 64 servers, (b) the traffic actually traverses that one degraded pair, and (c) the traffic is cross-rank, so it had to use the spine at all. In a rail-optimized topology, criterion (c) already excludes most of the load. In practice the oversubscription is invisible.

You keep all 64 servers and GPUs. You keep the symmetric uniform fat tree. You spend no extra hardware. You get two redundant fabric managers on physically distinct switches. The whole thing is a one-cable change.

Going to

Quantum-3 switches in the Q3400 family come with ports, so a leaf has ports facing GPUs and the natural cluster sizes land on GPUs instead of pure powers of two. The fabric-manager problem stays, though: with every port spoken for, you would still be sacrificing a server on some leaf, or a cable. As it turns out (thanks to Semianalysis for pointing this out), NVIDIA listened and added a dedicated UFM port to the switches.

At the same time, they disabled the possibility of running the subnet manager on managed Quantum-3 switches. In case you wonder whether IB forwarding across Ethernet might be possible, this unfortunately doesn’t work for control signals.

What “fair features” actually demand

Two desiderata for a feature map :

• Predictive signal. carries information about (otherwise we can’t use it for classification).
• Distributional insensitivity to the group attribute . That is, the group attribute shouldn’t matter in aggregate for the feature distribution. Operationally this gets split into two pieces:
  • Parity in the representation: .
  • Class-conditional separation: for each .

Without the second piece the goal collapses into a triviality: throw away all signal about and parity is automatic. Both together is the operational definition of a fair feature, and it is the form every paper above is chasing. The notation is the same as in the previous post: is the conditional mean embedding of the representation, and adds the class.

Linear algebra strikes again

Apply the law of total expectation conditional on . With class-conditional separation, the class-conditional means do not depend on , so we can drop the subscript and write . Then we can write the per-group mean as

Here is the group-specific base rate. Subtract across groups:

Parity zeros the left-hand side. Unequal base rates zero the scalar coefficient. The bracket must vanish:

In other words, the distributions for positive and negative classes match, since under a characteristic kernel on , equality of mean embeddings lifts to equality of distributions. We get . No measurable downstream predictor on retains any signal about .

The representation has been flattened. Both stacks of pancakes are now one.

Why this is harder to swallow than the Pokémon theorem

The Pokémon theorem said some fairness criterion remains unaudited after any finite checklist. There was a residual, and you could go bigger. The collapse here is not a residual. It is the joint distribution of being degenerate. Whatever your encoder is (linear, kernel, deep network, contrastive, adversarial), if it satisfies parity and class-conditional separation exactly and base rates differ, there is nothing in there about . Closely related observations appear in Lechner et al. (2021) and Zhao and Gordon (2022); the conjunction form here pins down the exact obstruction.

There is a decade of work that took the two desiderata as a starting point and built encoders that approximately satisfied both. The theorem does not say those encoders are useless. It only says that the limit they were chasing is empty (but there’s hope — see the next section).

The escape

The exact collapse only bites when both constraints hold exactly. Relax to -parity and -class-separation and you get that the usable class-conditional signal is bounded by

Linear in the joint fairness budget, rescaled by the base-rate gap. As the bound collapses to zero, matching the exact theorem. But for any positive budget there is a real Pareto frontier you can navigate and trade off fairness for usefulness in features.

The practical lesson is that the question is not whether your representation is fair. It is which fairness budget you spent and how much class signal you have left. Spend it deliberately.

The forbidden corner

The experimental check across three standard fairness benchmarks:

Each panel plots the parity gap on the horizontal axis against the largest class-conditional gap on the vertical axis. The line , with , is the theoretical bound from the approximate version of the theorem (Theorem 6.1 in the paper). The upper-left corner (small parity gap, large class-conditional signal) is the forbidden region.

The forbidden region is empirically empty. LFR, Fair-VAE, and adversarial debiasing across all three datasets cluster on the allowed side of the line. They are doing the only thing they can.

Closing

The Pokémon theorem said finite scalar checklists cannot catch every violation. The pancake theorem says perfect fair representations cannot retain . Two different orthogonality arguments, same RKHS engine. The paper has a Pokémon–KMR bridge and minimax rates for the approximate versions of both.

Paper: arXiv:2605.09221. Joint with Daniel Matsui Smola.

A Gaussian model of leaderboard scores

Let be the score matrix with models and benchmarks. Assume each row is an independent draw from a multivariate Gaussian:

Obviously wrong. Benchmark scores are bounded, often bimodal, frequently sparse. But it’s extremely useful: and are easy to estimate from leaderboard data (with EM for the missing entries you’ll inevitably have), and conditioning on a subset gives a closed-form imputer:

The conditional mean is the best linear predictor regardless of whether the scores are actually Gaussian, so the model doesn’t need to be right to give us a reasonable imputer. It just needs to give us .

This is the sensor placement problem

Once you have , picking which benchmarks to run is exactly the Gaussian process sensor placement problem Krause, Singh and Guestrin solved in 2008. Their sensors are our benchmarks; the field they’re trying to reconstruct is our matrix of unselected benchmark scores. Two natural objectives:

• Entropy . Pick benchmarks that are diverse from each other.
• Mutual information . Pick benchmarks that are maximally coupled with the ones you don’t run.

Both are submodular under the Gaussian model. Entropy also becomes monotone after rescaling the data so that every conditional variance exceeds — the rescaling only adds a modular term — so greedy comes with the standard approximation guarantee. MI is non-monotone in general but stays positive-margin for small in every dataset we tried, so we run greedy on it as a heuristic.

Greedy means: start empty, add the benchmark with the largest marginal gain, repeat. For entropy this is one rank-one Cholesky update per step; for MI it’s a fresh Cholesky on the complement block. Either way the total cost is negligible compared to running a single benchmark on a single model.

These two objectives pick different benchmarks. Entropy hunts for outlier subjects that disagree with everything else; MI hunts for hub subjects that strongly couple with the rest. For imputation, MI is what you want. But for diversity, entropy is the right tool. A surprise worth picking apart another time.

The numbers

We tried this on three score matrices: two from individual leaderboards, MMLU (5452 × 57, fully observed) and MTEB (263 × 56, 77% observed), plus a new one stitched together from nine public leaderboards (118 × 114, 31% observed). 10-fold CV, fit on 9 folds, impute the held-out fold.

• MMLU: . . Stable even with only 10% of models for training (~545 rows).
• MTEB: at , harder because the matrix is sparser and embedding tasks are genuinely heterogeneous.
• Merged: the hardest case ( at ), but still 15 benchmarks out of 114 explaining over half the variance in held-out scores.

A useful baseline: random selection on MMLU gets at . The 57 subjects are so redundant that you can almost throw darts. The gap opens up on MTEB and Merged, where the benchmark space has real structure and principled selection matters.

The hero image up top is what MI picks on MMLU, with selection positions averaged across the 10 folds (red diamonds) and per-fold positions in light blue. The first nine picks are identical across all 10 folds. The picks themselves are sensible: miscellaneous, professional_psychology, elementary_mathematics, high_school_psychology, marketing. A deliberately broad sweep of the MMLU domain.

We’re not alone

A few neighbors in this corner of the literature:

• tinyBenchmarks reduces redundancy within a benchmark by selecting representative examples.
• Sloth fits low-dimensional latent skills across benchmark families to predict performance.
• BenchBench diagnoses agreement among benchmarks via meta-benchmarking.
• BenchPress is the closest spiritual neighbor. It empirically observes that benchmark scores can be interpolated. Our entropy step turns out to be exactly pivoted Cholesky on , which is more or less what BenchPress does, but information-theoretically motivated rather than empirical.

One application: ranking a new benchmark

The same machinery applies in the dual. Instead of selecting benchmarks from a fixed pool, you can rank a new benchmark by its marginal MI gain against the incumbents. We tried this on the recently proposed ProactBench (arXiv:2605.09228), which decomposes conversational proactivity into Emergent, Critical, and Recovery axes. Recovery, assistance the user didn’t explicitly ask for, turns out to be weakly correlated with everything else ( vs. – for the standard benchmarks), and greedy entropy on the 9×9 correlation matrix ranks it #2 out of 9, ahead of every existing benchmark in the comparison. A clean example of when adding a new benchmark is actually worth the GPU time.

Paper: arXiv:2605.02209 · Workshop talk slides · Code and score matrices: github.com/smolix/benchmark-selection.

A kernel refresher

Pick a characteristic kernel on the feature space with feature map . For each group , the conditional mean embedding is

A linear mean-fairness criterion is a test direction , and the score (or classifier, or learned representation) passes the test exactly when the expected score on group matches that on group :

where is the group-difference vector. Demographic parity gaps, calibration moments, equalized-odds residuals, every “is the expected something the same across groups” check fits this form for some . A fairness checklist of size is a finite set of directions , and the classifier passes the checklist iff is orthogonal to every .

If the groups are distributionally distinct, , then by the characteristic property of . This is the only ingredient we need.

The geometric escape

Let be the subspace of the tests we apply. This is why the RKHS formulation helps: tests and distribution differences are vectors in the same inner-product space, so passing a mean-fairness criterion is exactly orthogonality to a test direction, and a whole checklist is just a subspace. If our classifier passes all criteria, then , the orthogonal complement of the audit subspace. Now consider the unit vector . By construction it also lies in , and

That direction is a fairness violation. We just constructed it. Whichever finite checklist you brought, the MMD witness is sitting in its orthogonal complement, perfectly visible, completely unaudited.

Gotta Catch ’Em All? Alas, the Pokémon theorem says that you can’t.

Why does this work?

Our argument is structural, not numerical. It does not depend on which criteria you chose, only on the fact that there are finitely many of them and the groups are distributionally distinct. The four-hundredth criterion buys you an audit subspace of dimension at most four hundred, and the orthogonal complement of a four-hundred-dimensional subspace of an infinite-dimensional Hilbert space is still infinite-dimensional. Some direction in it carries the group-difference signal. The MMD witness is always one such direction.

You can ask how large the residual is after criteria, and the paper gives a quantitative answer in terms of the spectral regularity of relative to the pooled data covariance: under polynomial eigendecay of the covariance operator and a source condition on , the minimax residual decays at the Kolmogorov -width rate . The minimax-optimal allocation of a size- fairness budget is the top- Mercer eigenspace of the pooled covariance. Spectral budgeting beats heuristic checklists. The paper has the details.

Provenance

I first posited this theorem (without proof) in a Stanford CS 329P lecture on applied machine learning. It sat there for a few years, in the form of a single slide and a feeling that it ought to be true. Then my son Daniel Matsui Smola and I sat down together and turned it into a proof, a quantitative version, and a handful of approximation results around it. I have written quite a few papers over the years. This is the first one I have written with Daniel, and I am incredibly proud of him. Watching him chase down the spectral arguments and add many more results has been one of the genuine joys this year.

Next post in this series: the impossibility of fair feature learning. A different and more brutal piece of orthogonality folklore from the same paper. Check out arXiv:2605.09221.