llm-service

Provider-agnostic LLM request and streaming handlers. One handler per provider (OpenAI, Anthropic, Google Gemini, xAI and Together AI) plus a router that picks the handler from the model name. Every handler returns the same LLMResponse shape.

The package also holds the server-side streaming pieces that both the local server (/llm/streamWithTTS) and the cloud functions use: NDJSON framing for LLM and combined LLM+TTS streams, the TTS adapter registry (OpenAI and Azure), the turn verdict, OpenAI embeddings and the effort mapping.

What’s in the package

Requests

import { handleLLMRequest } from 'llm-service';
 
const response = await handleLLMRequest({
  model: 'claude-haiku-5-5',
  input: [
    { role: 'system', content: 'Be brief.' },
    { role: 'user', content: 'Hello.' },
  ],
  max_tokens: 1024,
});
 
response.output_text;   // string
response.usage;         // { input_tokens, output_tokens, cached_input_tokens?, cache_creation_input_tokens? }
response.provider;      // 'anthropic'

handleLLMRequest calls getProviderForModel(model), which reads the provider from cortex’s model registry (getModelInfo), then calls the matching handler. The per-provider handlers handleAnthropicRequest, handleOpenAIRequest, handleGeminiRequest and handleOpenAIStreamRequest are also exported.

LLMRequest

LLMResponse carries output_text, usage, model, provider, finish_reason, latency_ms, optional top_logprobs ({ token, logprob }[]) and optional raw. usage.output_tokens includes reasoning tokens.

Streaming

import { handleLLMStreamRequest } from 'llm-service';
 
const { stream, aggregated } = handleLLMStreamRequest({
  model: 'claude-haiku-5-5',
  input: [{ role: 'user', content: 'Count to five.' }],
  max_tokens: 2048,
});
 
for await (const delta of stream) process.stdout.write(delta);
const final = await aggregated;   // LLMResponse with full usage

handleLLMStreamRequest is synchronous and returns an LLMStreamResponse: stream yields text deltas and aggregated resolves with the full response. LLMStreamRequest has the same model, input, max_tokens, temperature, effort and apiKey fields as LLMRequest, plus stop and text_format (a JSON schema for structured streaming output).

NDJSON orchestrators

Two helpers frame a stream as newline-delimited JSON on an HTTP response (Express and Firebase onRequest responses both fit NdjsonStreamResponse).

import { streamLlmToNdjson, streamLlmTtsToNdjson } from 'llm-service';
 
// Text only. Writes { t: 'delta', d } lines, then done.
await streamLlmToNdjson({ res, llmStream });
 
// Text plus audio.
const result = await streamLlmTtsToNdjson({
  res,
  llmStream,
  tts: (opts) => adapter.stream(opts),
  voice: 'en-US-AvaMultilingualNeural',
  firstChunkWordThreshold: 8,
  firstChunkTimeThresholdMs: 0,
});

The combined stream writes text, audio_start, audio (PCM16 24 kHz mono, base64), audio_end, audio_error, server_timing, llm_done, error and done frames (the NdjsonFrame union). If the client disconnects, the orchestrator stops writing and aborts TTS, but still drains the LLM stream so usage is complete. The result reports aggregated, totalTtsChars, ttsChunkCount, msTotal, aggregateOk and clientDisconnected.

Lower-level pieces are exported too: ResponseSplitter, wordCount, nthWordEndPosition, findBoundaryAfter, beginNdjsonStream, writeNdjsonLine, openaiTtsStream and TTS_TARGET_BYTES (6400).

Turn verdict

A small LLM call the host makes before the main model runs, to decide whether the user has finished speaking. The client sends a verdict block on the request body. The host writes the verdict frame before any delta and, on INCOMPLETE, ends the response without starting the main call.

import { parseVerdictRequest, runTurnVerdict, verdictFrame, writeNdjsonLine } from 'llm-service';
 
const verdict = parseVerdictRequest(req.body);   // null when absent or malformed
if (verdict) {
  const v = await runTurnVerdict({ input: req.body.input, verdict, receivedAtMs });
  writeNdjsonLine(res, verdictFrame(v));          // { t: 'verdict', complete, source, ... }
  if (!v.complete) return res.end();
}

Judge v1 shows the last 4 turns as chat turns and the sampled word decides. Judge v2 quotes the last 2 turns as a transcript, asks for top_logprobs and decides from P(COMPLETE) on the first token. It falls back to the word when the provider returns no probabilities (only Together returns them today).

runTurnVerdict never throws. A timeout, failure or unparsed answer counts as COMPLETE, and source says why: 'model', 'cap', 'error' or 'skipped' (no user utterance to judge). The result also carries timings (ms.queue, ms.model, ms.total), usage, raw, prob_score, utterance_words, context_turns and shown (what the judge was shown).

Also exported: sliceVerdictContext, toVerdictTurn, parseVerdict and VERDICT_SYSTEM_PROMPT. How the client uses the verdict is covered in Voice engine.

Effort

effort takes an EffortLevel from cortex: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh' or 'max'. A level must be one the model registry lists for that model. 'none' means thinking off. It is accepted for Anthropic, OpenAI and Together, and rejected for xAI and Gemini.

EffortAdapter is 'anthropic' | 'openai_responses' | 'openai_chat' | 'gemini' | 'xai' | 'together'.

TTS adapters

import { resolveAdapter, getProviderStatus, DEFAULT_TTS_PROVIDER } from 'llm-service';
 
const config = { openaiKey, azureKey, azureRegion };   // TtsAdapterConfig
const adapter = resolveAdapter(config, 'azure');        // ServerTtsAdapter | null
for await (const pcm of adapter!.stream({ text: 'Hi.', voice: 'en-US-AvaMultilingualNeural' })) { /* ... */ }

TtsProviderId is 'openai' | 'azure' and DEFAULT_TTS_PROVIDER is 'azure'. resolveAdapter falls back to the default when the requested provider can’t be built, and returns null when neither can. Adapters are built on first use and cached. getProviderStatus(config) lists each provider with available, isDefault and a reason when unavailable.

A ServerTtsAdapter has name, stream(opts) (PCM16 24 kHz mono buffers), listVoices() and estimateCost(opts). TtsStreamOpts takes text, voice, speed, instructions (OpenAI only), onTiming and signal.

Speed helpers: TTS_SPEED_RANGES (OpenAI 0.25 to 4.0, Azure 0.5 to 2.0), clampTtsSpeed(provider, speed), azureProsodyRate, buildAzureSsml and openaiSpeedInstruction.

Embeddings

import { openaiEmbedding, EMBEDDING_MODEL } from 'llm-service';
 
const { embedding, dimensions, inputTokens, costUsd } =
  await openaiEmbedding({ apiKey, text: 'hello', dimensions: 512 });

EMBEDDING_MODEL is text-embedding-3-small. Options are apiKey (required), text, dimensions and signal. Failures throw OpenAIEmbeddingError.

TTS token counting

OpenAI’s gpt-4o-mini-tts bills by input tokens. countGpt4oMiniTtsInputTokens(text) returns that count, and cortex exports the rates:

import { countGpt4oMiniTtsInputTokens } from 'llm-service';
import { GPT4O_MINI_TTS_PRICING } from 'cortex';
 
const tokens = countGpt4oMiniTtsInputTokens(text);
const inputCostUsd = tokens * GPT4O_MINI_TTS_PRICING.inputPricePer1M / 1_000_000;

Provider keys

Without apiKey, each handler reads its key from the environment: ANTHROPIC_API_KEY and OPENAI_API_KEY (through the SDK defaults), GEMINI_API_KEY, XAI_API_KEY and TOGETHER_API_KEY (or TOGETHERAI_API_KEY). The TTS registry does not read the environment. The host passes keys in TtsAdapterConfig.

See also

• cortex for the model registry and pricing
• Voice engine for how the app uses the turn verdict
• Source: packages/llm-service/src/index.ts