This is a voice agent.

You ask it to book a flight to Boston…

…then you change your mind. "No, sorry, New York."

Most agents act on the first thing they hear.

Ours waits until you finish.

Then it acts. Once.

Samsung PRISM · Theme 05 · Interruptible real-time agents

Commit Harness

A voice agent that acts only on what you finally meant.

Team Reign
  • Pokala Sai NithinTeam lead
  • Aryan GargOffline fallback
  • LohitashwaReproduction, harness rules
  • V PreethaDocumentation & Media Lead
College
SRM Institute of Science and Technology
Voice model
Gemini 3.8 Live
Benchmark
Full-Duplex-Bench v3
100 real recordings
Timeline
29 to 30 Sep 2026
Description
A small layer between the voice model and its tools. It holds each action until you finish speaking, replaces it when you correct yourself, cancels it when you say "never mind", and runs it once.
Context
People pause, hesitate and correct themselves while they talk. A voice agent that acts at the first pause books the wrong flight, then books a second one. The benchmark counts every action that really runs, so one early action fails the whole request.

Propose, settle, commit

Gemini 3.8 Live listens and speaks. When it wants to act, it proposes a tool call (an action, such as "track this order"). The harness decides when that call can run. Pick a request below and watch what the harness does.

    An illustration of the four rules. The real decisions of every run are in gate_events.log.

    Who decides that you finished

    Reflex reads word patterns such as "um" or "no, sorry" and costs nothing. It waits for 0.9 s of quiet, or 1.8 s when you sound unsure. Reasoner (TypeSafe Jev, a small classifier) judges the meaning. If Reasoner takes longer than 0.8 s, Reflex decides alone. A third decider, Smart Turn, hears the tone of the voice. It was wrong too often, so it is off in the submitted run.

    Two rules around the harness also matter. The prompt rules tell the model that the last value wins. The identifier rule joins "B-O-B-1-2" into "BOB12" before the tool runs.

    The benchmark

    Full-Duplex-Bench v3 plays 100 recordings of real people who ask for actions out loud, in shopping, finance, housing and travel. The strict score counts only calls that are exactly right. The judged score lets a language model accept calls that are right in meaning. Our judge is Gemini 2.5 Pro with the benchmark's own prompts, in place of GPT-4o, the same for every agent.

    Our agent, judged
    67 / 100
    55 strict · 5.3 s typical reply
    Gemini 3.8 Live alone, judged
    62 / 100
    50 strict · 3.9 s typical reply
    Judge

    Gemini 2.5 Pro, with the benchmark's own judge prompts, unchanged. The official judge is GPT-4o. We had no access to it, and we used no OpenAI model anywhere. Both agents had the same judge. A GPT-4o score can differ by a few recordings.

    StrictJudged
    Gemini alone
    50
    62
    Our run, 29 Sep
    46
    61
    Submitted
    55
    67
    Smart Turn on
    50
    64

    Passed recordings out of 100. Judge: Gemini 2.5 Pro. One run each. A teammate repeated our run on his own laptop and got 53 strict.

    Where the points came from

    Gemini aloneOur agent
    Shopping (29)
    22
    24
    Finance (25)
    22
    22
    Housing (26)
    5
    7
    Travel (20)
    13
    14
    Three requests in one turn (16)
    5
    7
    Self-corrections (17)
    8
    7

    Judged passes in each slice. The bar shows the share of the slice.

    What the logs show

    The harness writes every decision to a log, so we counted how often it changed what ran. In one full run, Gemini proposed 148 calls:

    146
    ran unchanged
    1
    replaced after a correction
    1
    duplicate blocked
    +0.9 s
    wait before each call

    Gemini usually waits until it thinks you have finished, so the harness rarely needs to step in. Our gain came from the prompt rules and the identifier rule, which we switched on together. Ten failures were changes of mind after the call had already run. No waiting rule can fix those. They need undo, which the extension below provides.

    How we compare

    Three labs now publish voice agents built the way ours is: a fast model talks and listens, and a slower model or layer does the tool work. The chart shows the published Full-Duplex-Bench v3 tool-call scores. Each lab ran its own setup and its own judge, so read the bars as a range, not as one leaderboard.

    Reported by the labOurs, judgedOurs, strict
    GPT-Live-1 with a GPT backend
    87.0
    Thinking Machines interaction model
    68.0
    Our agent on Gemini 3.8 Live
    67
    55
    Gemini 3.8 Live alone
    62
    50
    GPT-Realtime-2.1
    60.0
    GPT-Realtime-2
    58.0

    Full-Duplex-Bench v3, tool-call pass rate out of 100. GPT rows: OpenAI's GPT-Live-1 API launch post (backend "Terra, low"). Thinking Machines: Interaction models, 11 May 2026, research preview. Our rows: one run each, judge Gemini 2.5 Pro in place of GPT-4o.

    What each system tells you

    The scores show who calls the right tool. They do not show what happens when a slow tool is still running and the user changes their mind. That is the part we built, so the table below lists only what each team has published about it.

    GPT-Live-1Thinking MachinesKyutai Moshi (RL)Our agent
    Fast talker, slow thinkerYes, delegates to a backend modelYes, a background modelNo, one modelYes, Gemini Live plus our Thinker
    Calls toolsYesYesNoYes
    Waits until the user finishesNot publishedNot publishedTalking onlyYes, holds each call until the turn settles
    Drops a stale result after a correctionNot publishedTimes when to speak a resultNo toolsYes, by plan version
    Undoes an action that already ranNot publishedNot publishedNo toolsYes, newest step first
    Hands off after repeated failuresNot publishedNot publishedNo toolsYes, after two failures
    Works on any voice modelOpenAI onlyOwn modelOpen weightsYes, a layer outside the model
    AvailablePaid APIResearch previewFree, 8B weightsOpen source

    "Not published" means that we found no public description, not that the feature is missing. Kyutai's models target Full-Duplex-Bench v1 and v2 (talking manners) and do not call tools. ByteDance SeedRealtime is left out: it is available only in the Doubao app and publishes no benchmark.

    What we ran into, and why we built it this way

    We started with a simple idea and met six limits. Each limit shaped one choice in the design. This section lists them in order.

    1. The best voice models are closed

    We cannot change the inside of Gemini Live. Thinking Machines released its interaction model in May 2026 as a research preview with closed weights, and it has no public API. ByteDance SeedRealtime runs only inside the Doubao app and has no developer API. NVIDIA PersonaPlex-7B is open, but its authors list tool calling as a future step, so it talks and does not act. Because we could not retrain any of these models, we put the safety work in a layer outside the model. That layer works with any voice model that sends tool calls, and this is the reason we call it a harness.

    2. The benchmark tools never fail

    The benchmark tools answer at once and are always right. Real tools are slow, fail, and run twice. So we added a recovery layer with mock tools. It tries a failed call again, blocks a repeated booking, drops a result that comes back late, and undoes finished steps, newest first. We did not have a real scheduling system, so every claim about these tools is about mock tools.

    3. Speech is messy

    People pause inside a sentence and then change their mind. A model that acts on the first words books the wrong thing. This is the reason for the core rule: hold each call until the turn settles, then act once. On real SLURP recordings with a pause added, an agent without the harness went from 10 right out of 11 to 5 right, and 6 of 11 did a wrong or extra action.

    4. Money needs a clear yes

    In the care demo, a payment must never go through on a loose "okay". The tool now refuses to charge the card until the customer says yes to the exact fee that the agent just asked about. A review found a hole: an "okay" to a later question, such as a text reminder, could count as consent. We closed it. Any other question from the agent now cancels the open fee question, and the agent must ask again. One live check is still open. We must confirm that the real agent's spoken question reaches this check.

    5. We needed a fast talker and a slow thinker

    The two best published systems use two models. A fast model keeps the conversation going. A slower model does the multi-step work. We built the same shape on Gemini. Gemini Live talks, and an optional second model, the Thinker, plans the tool work. The Thinker is off by default, because the live care demo works with Gemini Live's own tool calls, and we have not run the Thinker live.

    6. The cloud can disappear, and we had no GPU

    A car or a home agent must work with no connection. We tested two Gemma 4 models on a laptop with 32 GB of memory and no graphics card. The large one was right on 45 of 51 real light requests, and the small one on only 18. We chose the large one, at about 6 seconds a reply.

    Why this approach, and where it falls short

    We chose a layer outside the model for three reasons. It works on models we cannot open. It is easy to test, because the tests use a fake clock and mock tools and need no keys. And it gives a proof at the end: a check of the final state shows one booking, one charge, and no hold left over. It also has limits. Thinking Machines scores higher on the benchmark, with 68.0 against our 67 judged and 55 strict. Their model also acts on its own, for example on timers and video cues, and ours speaks only after the customer speaks or a tool answers. Our scores come from one run each with a different judge, so read them as a range. We built our layer for tool safety. We did not build a new model.

    Facts about other systems come from public write-ups and may change. We did not check them with the labs.

    In the car: when tools fail

    The benchmark's tools answer at once and never fail. Real tools do not. So we built a recovery layer for an in-car assistant for an electric car, with mock tools. It tries a failed call again, blocks a repeated booking, cancels an old booking before it makes a new one, and passes you to a human after two failures. Step through the recorded drive:

    EV assistant · recovery layerrecovery log
    Driver
    Assistant

    Played through LiveKit as recorded audio, the way the benchmark plays its recordings. Mock tools, synthetic voice, one run.

    On real voices

    To test real speech, we used 11 recordings from SLURP, a public dataset of real people's requests to a home assistant (Bastianelli et al., EMNLP 2020). The lights tool failed on its first try every time.

    10/11
    requests carried out
    8/8
    failures recovered by a retry
    1
    bug found and fixed

    When the speaker pauses mid-sentence

    Then we put a silence of 1.6 to 3.1 s inside each of the same requests. This agent has the recovery layer but not the harness, and the pauses broke it.

    10 → 5
    fully right, of 11
    0 → 6
    wrong or extra action, of 11

    "Turn the lights off."

    2.6 s pause inside

    Turned the lights on.

    "Light colour for study room."

    2.8 s pause inside

    Set the study AC to 22.

    "Turn my lights down to a lower level of brightness."

    3.1 s pause inside

    Dimmed them, then rang the phone.

    The two layers must work together in one agent. That is our next step.

    Offline: when the cloud is gone

    If the car loses its connection, a local model must take over. Our teammate Aryan tested two Gemma 4 models with Ollama on a laptop with 32 GB of memory and no graphics card. Each request was a written sentence, the text that the model gets after speech-to-text.

    Gemma 4 26B · 15.9 GB · ~6 sGemma 4 e4b · 3.1 GB · ~2 s
    Our 40 requests
    34/36
    34/36
    Real SLURP light requests
    45/51
    18/51
    No fitting tool, did nothing
    60/60
    60/60
    Self-corrections
    28/29
    29/29
    "Never mind", did nothing
    8/8
    6/8

    The small model refused most real requests, and after "leave them as they are" it still turned the lights off. We chose the large one.

    Try it

    You need Linux or WSL, a LiveKit Cloud project and a Gemini API key. The script asks for the keys on the first run. A full run of 100 recordings takes about 2 hours.

    git clone https://github.com/Nithyon/Theme5-Interruptible-Agents
    cd Theme5-Interruptible-Agents
    ./reproduce.sh                       # the benchmark agent
    ./reproduce_extension.sh             # recovery layer tests, no keys
    ./reproduce_extension.sh fallback    # the Gemma fallback, needs Ollama

    Every run we report has its logs in project-log/runs/, down to each recording.