Commit Harness
A voice agent that acts only on what you finally meant.
- Description
- A small layer between the voice model and its tools. It holds each action until you finish speaking, replaces it when you correct yourself, cancels it when you say "never mind", and runs it once.
- Context
- People pause, hesitate and correct themselves while they talk. A voice agent that acts at the first pause books the wrong flight, then books a second one. The benchmark counts every action that really runs, so one early action fails the whole request.
Propose, settle, commit
Gemini 3.8 Live listens and speaks. When it wants to act, it proposes a tool call (an action, such as "track this order"). The harness decides when that call can run. Pick a request below and watch what the harness does.
An illustration of the four rules. The real decisions of every run are in gate_events.log.
Who decides that you finished
Reflex reads word patterns such as "um" or "no, sorry" and costs nothing. It waits for 0.9 s of quiet, or 1.8 s when you sound unsure. Reasoner (TypeSafe Jev, a small classifier) judges the meaning. If Reasoner takes longer than 0.8 s, Reflex decides alone. A third decider, Smart Turn, hears the tone of the voice. It was wrong too often, so it is off in the submitted run.
Two rules around the harness also matter. The prompt rules tell the model that the last value wins. The identifier rule joins "B-O-B-1-2" into "BOB12" before the tool runs.
The benchmark
Full-Duplex-Bench v3 plays 100 recordings of real people who ask for actions out loud, in shopping, finance, housing and travel. The strict score counts only calls that are exactly right. The judged score lets a language model accept calls that are right in meaning. Our judge is Gemini 2.5 Pro with the benchmark's own prompts, in place of GPT-4o, the same for every agent.
Gemini 2.5 Pro, with the benchmark's own judge prompts, unchanged. The official judge is GPT-4o. We had no access to it, and we used no OpenAI model anywhere. Both agents had the same judge. A GPT-4o score can differ by a few recordings.
Passed recordings out of 100. Judge: Gemini 2.5 Pro. One run each. A teammate repeated our run on his own laptop and got 53 strict.
Where the points came from
Judged passes in each slice. The bar shows the share of the slice.
What the logs show
The harness writes every decision to a log, so we counted how often it changed what ran. In one full run, Gemini proposed 148 calls:
Gemini usually waits until it thinks you have finished, so the harness rarely needs to step in. Our gain came from the prompt rules and the identifier rule, which we switched on together. Ten failures were changes of mind after the call had already run. No waiting rule can fix those. They need undo, which the extension below provides.
How we compare
Three labs now publish voice agents built the way ours is: a fast model talks and listens, and a slower model or layer does the tool work. The chart shows the published Full-Duplex-Bench v3 tool-call scores. Each lab ran its own setup and its own judge, so read the bars as a range, not as one leaderboard.
Full-Duplex-Bench v3, tool-call pass rate out of 100. GPT rows: OpenAI's GPT-Live-1 API launch post (backend "Terra, low"). Thinking Machines: Interaction models, 11 May 2026, research preview. Our rows: one run each, judge Gemini 2.5 Pro in place of GPT-4o.
What each system tells you
The scores show who calls the right tool. They do not show what happens when a slow tool is still running and the user changes their mind. That is the part we built, so the table below lists only what each team has published about it.
| GPT-Live-1 | Thinking Machines | Kyutai Moshi (RL) | Our agent | |
|---|---|---|---|---|
| Fast talker, slow thinker | Yes, delegates to a backend model | Yes, a background model | No, one model | Yes, Gemini Live plus our Thinker |
| Calls tools | Yes | Yes | No | Yes |
| Waits until the user finishes | Not published | Not published | Talking only | Yes, holds each call until the turn settles |
| Drops a stale result after a correction | Not published | Times when to speak a result | No tools | Yes, by plan version |
| Undoes an action that already ran | Not published | Not published | No tools | Yes, newest step first |
| Hands off after repeated failures | Not published | Not published | No tools | Yes, after two failures |
| Works on any voice model | OpenAI only | Own model | Open weights | Yes, a layer outside the model |
| Available | Paid API | Research preview | Free, 8B weights | Open source |
"Not published" means that we found no public description, not that the feature is missing. Kyutai's models target Full-Duplex-Bench v1 and v2 (talking manners) and do not call tools. ByteDance SeedRealtime is left out: it is available only in the Doubao app and publishes no benchmark.
What we ran into, and why we built it this way
We started with a simple idea and met six limits. Each limit shaped one choice in the design. This section lists them in order.
1. The best voice models are closed
We cannot change the inside of Gemini Live. Thinking Machines released its interaction model in May 2026 as a research preview with closed weights, and it has no public API. ByteDance SeedRealtime runs only inside the Doubao app and has no developer API. NVIDIA PersonaPlex-7B is open, but its authors list tool calling as a future step, so it talks and does not act. Because we could not retrain any of these models, we put the safety work in a layer outside the model. That layer works with any voice model that sends tool calls, and this is the reason we call it a harness.
2. The benchmark tools never fail
The benchmark tools answer at once and are always right. Real tools are slow, fail, and run twice. So we added a recovery layer with mock tools. It tries a failed call again, blocks a repeated booking, drops a result that comes back late, and undoes finished steps, newest first. We did not have a real scheduling system, so every claim about these tools is about mock tools.
3. Speech is messy
People pause inside a sentence and then change their mind. A model that acts on the first words books the wrong thing. This is the reason for the core rule: hold each call until the turn settles, then act once. On real SLURP recordings with a pause added, an agent without the harness went from 10 right out of 11 to 5 right, and 6 of 11 did a wrong or extra action.
4. Money needs a clear yes
In the care demo, a payment must never go through on a loose "okay". The tool now refuses to charge the card until the customer says yes to the exact fee that the agent just asked about. A review found a hole: an "okay" to a later question, such as a text reminder, could count as consent. We closed it. Any other question from the agent now cancels the open fee question, and the agent must ask again. One live check is still open. We must confirm that the real agent's spoken question reaches this check.
5. We needed a fast talker and a slow thinker
The two best published systems use two models. A fast model keeps the conversation going. A slower model does the multi-step work. We built the same shape on Gemini. Gemini Live talks, and an optional second model, the Thinker, plans the tool work. The Thinker is off by default, because the live care demo works with Gemini Live's own tool calls, and we have not run the Thinker live.
6. The cloud can disappear, and we had no GPU
A car or a home agent must work with no connection. We tested two Gemma 4 models on a laptop with 32 GB of memory and no graphics card. The large one was right on 45 of 51 real light requests, and the small one on only 18. We chose the large one, at about 6 seconds a reply.
Why this approach, and where it falls short
We chose a layer outside the model for three reasons. It works on models we cannot open. It is easy to test, because the tests use a fake clock and mock tools and need no keys. And it gives a proof at the end: a check of the final state shows one booking, one charge, and no hold left over. It also has limits. Thinking Machines scores higher on the benchmark, with 68.0 against our 67 judged and 55 strict. Their model also acts on its own, for example on timers and video cues, and ours speaks only after the customer speaks or a tool answers. Our scores come from one run each with a different judge, so read them as a range. We built our layer for tool safety. We did not build a new model.
Facts about other systems come from public write-ups and may change. We did not check them with the labs.
In the car: when tools fail
The benchmark's tools answer at once and never fail. Real tools do not. So we built a recovery layer for an in-car assistant for an electric car, with mock tools. It tries a failed call again, blocks a repeated booking, cancels an old booking before it makes a new one, and passes you to a human after two failures. Step through the recorded drive:
Played through LiveKit as recorded audio, the way the benchmark plays its recordings. Mock tools, synthetic voice, one run.
On real voices
To test real speech, we used 11 recordings from SLURP, a public dataset of real people's requests to a home assistant (Bastianelli et al., EMNLP 2020). The lights tool failed on its first try every time.
When the speaker pauses mid-sentence
Then we put a silence of 1.6 to 3.1 s inside each of the same requests. This agent has the recovery layer but not the harness, and the pauses broke it.
"Turn the lights off."
2.6 s pause inside
Turned the lights on.
"Light colour for study room."
2.8 s pause inside
Set the study AC to 22.
"Turn my lights down to a lower level of brightness."
3.1 s pause inside
Dimmed them, then rang the phone.
The two layers must work together in one agent. That is our next step.
Offline: when the cloud is gone
If the car loses its connection, a local model must take over. Our teammate Aryan tested two Gemma 4 models with Ollama on a laptop with 32 GB of memory and no graphics card. Each request was a written sentence, the text that the model gets after speech-to-text.
The small model refused most real requests, and after "leave them as they are" it still turned the lights off. We chose the large one.
Try it
You need Linux or WSL, a LiveKit Cloud project and a Gemini API key. The script asks for the keys on the first run. A full run of 100 recordings takes about 2 hours.
git clone https://github.com/Nithyon/Theme5-Interruptible-Agents
cd Theme5-Interruptible-Agents
./reproduce.sh # the benchmark agent
./reproduce_extension.sh # recovery layer tests, no keys
./reproduce_extension.sh fallback # the Gemma fallback, needs Ollama
Every run we report has its logs in project-log/runs/, down to each recording.