Voice engine

In a voice session, SmartChats has to decide when you have finished speaking and your words should go to the agent. The part that makes this decision is vie, the voice interface engine (packages/vie). It is the default engine. The older pipeline stays available as voiceEngine: 'legacy'.

This page explains how vie works, what you can change, and where to look when a turn ends too early or too late. The full design is in the vie spec.

What the engine does

vie reads four kinds of signal:

• Audio power: is the room loud enough to start the speech recognizer?
• The speech detector (VAD): when speech starts and stops.
• The browser speech recognizer: the text, updated word by word.
• A judge: does the text so far sound finished?

From these it produces a fixed chain of events for each turn:

listen.wanted → speech.ended → transcript.settled → turn.sent → turn.ended → turn.shipped → (turn.amended)

Around that chain, the turn lifecycle adds four more events:

Only one thing is ever undone. If speech starts again while vie waits for the settle, speech.ended is cancelled and the turn carries on as one.

The turn states look like this:

listening ─ vad.end ─► ended ─ settleMs quiet ─► judging ─ COMPLETE ───► shipped ─ reply done ─► idle
    ▲                    │                          │
    │      vad.start ────┘ (the one retraction)     ├─ INCOMPLETE ─► held ─ silence / keyword ─► shipped (as is)
    │                                               │                 │
    └──────────── continuation ◄────────────────────┴─ abort ─► held ─┘ new words ─► merged into the turn

The three judges

The speech detector and the settle only know that you paused. The judge decides whether the pause ended your thought. Choose one in Settings under “Who decides you’re done talking” (tivi setting vieJudge).

Every judge goes through the same judging state. They differ only in how they answer:

• Server: vie sends the text with a verdict request at the settle (turn.sent). The verdict arrives before any reply text, usually 0.3 to 0.5 s later. If it never arrives, the turn is confirmed after 3 s.
• Smart Turn: vie scores the audio. A probability at or above threshold is COMPLETE. Past judgeTimeoutMs (1.4 s) the turn ships anyway.
• None: COMPLETE at once.

Words that go to a tool waiting for input (dictation, voice memo) always use no judge, because they make no model call for the server to judge.

The server turn verdict

The server judge is one small LLM call made before the main reply. It answers COMPLETE (the reply proceeds) or INCOMPLETE (the turn is held). The code is in packages/llm-service/src/streaming/verdict.ts, and TURN_VERDICT.md has the measurements.

The app asks for a verdict only when all of these hold:

• the call comes from a running voice session
• the voice engine is vie and the judge is server
• a verdict model is set (verdictModel, default deepseek-v4.1-flash)

Judge v2

The app sends judge: 'v2'. Under judge v2:

• The score is P(COMPLETE), the model’s own probability that its one-word answer is COMPLETE. It is read from the first output token’s probabilities, not written by the model.
• The threshold defaults to 0.5. The server holds the turn when the score is below it.
• The input is the last 2 conversational turns, quoted as a transcript (“User: … / Assistant: …”) in one message. The judge reads the exchange as a third party. This stops a reply like “go ahead, I’m listening” from reading as an invitation to keep talking.

Scores pile up at both ends. Fewer than 3% fall between 0.1 and 0.9, so the threshold matters little in practice. There is no tuner for it.

Holds and merges

When the verdict is INCOMPLETE:

1. The server ends the response without making the main model call.
2. The app plays the keep-going cue. This is a short tone, or a spoken “ok” / “mhm” when acks are on (enableAcks).
3. Whatever you say next is merged with the held text and sent as one message (turn.merged).
4. If you say nothing for holdSilenceMs (3 s), or say a commit keyword (“done”, “finished”, “listo”, “terminado”), the held text ships as it is (turn.forced). A forced turn carries no verdict.

Providers

Judge v2 needs a provider that returns token probabilities. Today that is Together, so the default verdict model needs a Together key.

• With a Together key, v2 decides from P(COMPLETE).
• With a verdict model from another provider, v2 decides from the sampled word and the threshold has no effect.
• With no key for the verdict model, the local server skips the verdict and answers the turn, as if the verdict had said COMPLETE.

See Providers & API keys for how to set the key.

Parameters

All timing lives in packages/vie/src/core/params.ts.

How tuning works

vie learns its timing on each device from what happens in real turns:

• Settle eases down and jumps up. On every settle, settleMs moves 25% of the way toward the largest gap between word updates in that settle window. These moves are quiet. When a word lands late after a ship, settleMs jumps straight to that word’s gap. On a clean device settle drifts down toward the recognizer’s real gaps, jumps when a word arrives late, then eases down again.
• The tail is fixed at 200 ms. Deciding whether a pause ended your thought is the settle’s and the judge’s job.
• Pre-roll and the Smart Turn threshold ratchet. Each moves at most one step per turn, and only in its safe direction: pre-roll up after a clipped first word, the threshold down after a wrong hold. They do not come back down on their own.
• Pins. A pinned parameter never moves. Moving a slider in Settings pins it.
• Per device. Parameters, pins, the evidence ledger and the tuner history are stored in the browser’s localStorage under vie.device.v1. A phone and a laptop each learn their own values.

Continuations, late words and amends

Two things can happen after a turn has settled. vie tells them apart by whether the speech detector heard new speech.

Late words. A word update that arrives after the settle with no new speech start is the recognizer finishing what it already heard. It belongs to the same utterance, in any phase:

• While the judge runs, the words are added to the turn. For the server judge the request is sent again with the full text.
• While the turn is held, they are added to the held text.
• After the ship, the turn is amended (turn.amended). Before the reply’s first token, the app cancels the reply and sends one complete message in place of the old one. After that, the words are handed to the agent as late words.

A recognizer revision is not late words. If the recognizer only repeats the end of what was already sent (for example by dropping a word), the sent text is kept and only the new words after it count.

Continuations. New speech after the settle, a speech start followed by its first word, is a continuation. vie acts at the first word, not the bare speech start, because coughs and background talk often start the detector without producing words.

Settings you can change

These are tivi settings (packages/tivi/src/lib/settings.ts), set in the Settings panel:

The engine choice and the verdict settings are only in the Settings panel. The agent can change the rest by voice. Ask it, for example, “turn on backchannel acks” or “pin the settle time at 500 ms”. Its tools:

• set_voice_setting and reset_voice_settings for settings such as acks, pre-roll and playback speed
• get_voice_tuning for current parameters, pins, recent tuner moves and device alerts
• pin_voice_setting, unpin_voice_setting, reset_voice_tuning and set_auto_tune for vie’s learned parameters
• report_speech_cutoff when a transcript lost its first word (this raises pre-roll). A lost last word is recorded but does not move settle.

/voicelab shows the engine’s layers live and runs calibration.

Telemetry

vie writes these events to insights:

A healthy device shows settle easing down, an occasional jump on a late word, and a low rate of late words. Each late word cancels a reply in progress.

See also

• vie spec: states, tuning rules, invariants and open items
• TURN_VERDICT.md: the server verdict, judge v2 and the replay script for measuring a prompt or model change
• Providers & API keys
• tivi