Architecture

This page gives an overview of the whole system first, then goes into the data layer, which most contributions touch.

Request flow

The browser app does most of the work: it runs the agent loop, the voice engine and the code sandbox. A server (the local server when you self-host, the hosted cloud otherwise) holds the API keys, calls the model providers, and owns the database and media storage.

 browser: SmartChats app
 ┌───────────────────────────────────────────────┐
 │ mic ─► tivi (VAD, speech recognition)         │
 │          │                                    │
 │          ▼                                    │
 │        vie (turn-taking) ── turn ready ──┐    │
 │                                          ▼    │
 │        cortex (agent loop) ◄── typed text     │
 │          │                                    │
 │          ▼                                    │
 │        SmartChatsBackend (local or cloud)     │
 └──────────┬────────────────────────────────────┘
            │ HTTP, NDJSON stream back
            ▼
 server: smartchats-local-server (/local-api/*)
 ┌───────────────────────────────────────────────┐
 │ /llm/streamWithTTS                            │
 │   1. turn verdict (small model): done talking?│
 │        INCOMPLETE ─► end response, keep       │
 │                      listening                │
 │   2. main model ─► llm-service providers      │
 │   3. sentences ─► TTS ─► audio frames         │
 │ /data  ─► SurrealDB                           │
 │ /storage ─► S3 bucket (e.g. SeaweedFS)        │
 │ /tools ─► web search, URL fetch               │
 └───────────────────────────────────────────────┘

The pieces

The app and its shells

apps/smartchats is a Next.js app built as a static single-page app. Its UI is a set of shells that share one store: the default desktop shell and a mobile shell, picked in settings, plus shells mounted on their own routes, /sail and /voicelab. Built-in mini apps (log, metrics and knowledge-graph explorers, todos and others) live in app/apps/. The app talks to its server only through the SmartChatsBackend interface described below.

cortex: the agent loop

packages/cortex is the agent framework. It composes the system prompt, calls the model through the backend, parses structured output, runs the functions and code the model asks for in a sandbox, and keeps workspace state across turns. It runs in the browser and in Node. Hosts inject the sandbox and the transport. It also holds the model catalog, presets and voices.

llm-service: providers, streaming and the turn verdict

packages/llm-service is the provider layer used by both servers. It normalizes OpenAI, Anthropic, Gemini, xAI and Together AI into one request and stream shape, with structured output, effort levels and bring-your-own keys. Text-to-speech adapters cover OpenAI and Azure.

The combined LLM and TTS handler streams NDJSON: text deltas, then audio for each sentence as soon as it is ready. When a voice request carries a verdict block, the handler first makes one small model call that judges whether the user has finished speaking, and writes that verdict frame before any delta. On INCOMPLETE it ends the response without starting the main call. See Providers.

tivi and vie: voice in and turn-taking

packages/tivi provides the voice interface as React hooks: voice activity detection, browser speech recognition, the TTS playback queue and calibration. packages/vie is the voice engine and the default turn-taking engine. It decides when a turn has ended from VAD and recognizer signals, asks a judge (the server verdict by default, or the local Smart Turn model), and tunes its timing per device from logged evidence. The older engine stays behind the voiceEngine: 'legacy' setting. See Voice engine.

smartchats-local-server

packages/smartchats-local-server is the self-hosted Express server. It serves the app and the API under /local-api/*: health, llm, tts, embeddings, data, usage, keys, tools, insights and storage. Data lives in SurrealDB. Media uploads go straight from the browser to an S3-compatible bucket through signed links, with a media table tracking each object. smartchats start runs SeaweedFS as that bucket and sets the SMARTCHATS_S3_* variables, or you point them at your own. See Local server.

smartchats-hub, the relay and the MCP bridge

These let coding agents talk with you through the app. smartchats-relay-cf is the WebSocket relay between agents and your browser, partitioned by user. smartchats-relay-protocol defines its message envelope and smartchats-relay-connect handles connect, reconnect and re-auth. smartchats-mcp-bridge is an MCP server one agent runs to reach the relay. smartchats-hub is a per-machine daemon that holds one relay connection for every local agent, refreshes the token once, keeps a durable inbox per agent and serves MCP on 127.0.0.1. See Hub and Relay.

smartchats-tools

packages/smartchats-tools holds the agent’s I/O helpers, such as web search through Serper and URL fetch with readable-text extraction. They are pure functions with no host dependencies, so the local server’s /tools/* routes and the cloud use the same code.

MCP

packages/smartchats-mcp exposes the data layer to MCP clients over stdio. See MCP Server.

The data layer

The data layer follows three rules:

1. One contract for data access. Every consumer (browser app, CLI, MCP server) talks to data through the same DataAPI interface.
2. Two implementations. Cloud (your smartchats.ai account) and local (your own install). Both implement the same contract.
3. One home for each kind of change. Query builders, multi-step operations and schema changes each have a set place, and consumers pick them up.

A new feature is usually one small change in one place. If one feature touches five files, look for the existing home first.

Layers

                    smartchats web app                                 what users see
                            │
                            ▼
                  getBackend().data.query(...)
                            │
       ┌────────────────────┴──────────────────┐
       │                                       │
   smartchats-cli                       smartchats-mcp                  other consumers
       │                                       │
       └────────┬──────────────────────────────┘
                ▼
    operations.importBundle / exportBundle                              multi-step ops
                ▼
     queries.X(args) → { query, variables }                             pure builders (no I/O)
                ▼
                DataAPI.query                                           contract (smartchats-backend)
              ╱            ╲
   makeCloudDataAPI       makeLocalDataAPI                              factories (smartchats-database)
        │                       │
   cloud-client              createClient
   (hosted cloud)            (your local SurrealDB)

Adding a query the agent can run

Three files, about 20 lines.

1. Add a builder in packages/smartchats-database/src/queries/your_topic.ts:

   export function findRecentByTag(args: { tag: string; limit?: number }): QuerySpec {
     return {
       query: 'SELECT * FROM logs WHERE tags CONTAINSALL [$tag] ORDER BY ts DESC LIMIT $limit',
       variables: { tag: args.tag, limit: args.limit ?? 20 },
     };
   }

2. Expose it to MCP in packages/smartchats-mcp/src/tools.ts:

   server.tool(
     'find_recent_by_tag',
     'Find recent logs tagged with a given label',
     { tag: z.string(), limit: z.number().optional() },
     async (args) => runAndFormat(queries.findRecentByTag(args), handle),
   );

3. Expose it to the CLI as a subcommand in packages/smartchats-cli/src/commands/:

   const result = await handle.data.query(queries.findRecentByTag(args));
   console.log(JSON.stringify(result.rows, null, 2));

The same builder works against the cloud (after smartchats login) and against a local install. The builder has no transport-specific code.

Adding a multi-step operation

If the work is more than one query, for example looping over a bundle and writing each row with progress and error reporting, put it in operations/:

// packages/smartchats-database/src/operations/your_op.ts
import type { DataAPI } from 'smartchats-backend';
 
export async function yourOp(data: DataAPI, args: YourOpArgs): Promise<YourOpResult> {
  // loops, error aggregation, progress callbacks
  await data.query(...);
}

The CLI and MCP then both call operations.yourOp(handle.data, args). See import_bundle.ts and export_bundle.ts for real examples.

The facade pattern in practice

Two interfaces wrap the backends.

1. SmartChatsBackend (in the web app)

The web app uses the full SmartChatsBackend interface from packages/smartchats-backend/src/types.ts. Besides data it has llm, tts, embeddings, usage, keys, billing, tools, integrations, insights and storage, plus capabilities, health() and providers().

// apps/smartchats/src/lib/backend_facade.ts
const backend = getBackendInstance();
 
await backend.llm.streamWithTTS(...);         // chat completion with speech
await backend.tts.stream(...);                // text-to-speech
await backend.embeddings.embed(...);          // embedding vectors
await backend.data.query(...);                // SurrealDB read/write
await backend.storage.createUpload(...);      // media upload links
await backend.insights.emit(events);          // telemetry
const health = await backend.health();        // aggregate health probe
const report = await backend.providers();     // which LLM / TTS providers can serve

BackendFacadeProvider puts the active backend into a React context. Features check capabilities: a self-hosted backend reports billing: false, so the credit chip hides, the billing page redirects and the bring-your-own-key UI shows instead. storage is false until a bucket is configured. providers() tells the model menu which models and voices have a key behind them.

2. DataAPIHandle (in the CLI and MCP)

The CLI and MCP only need data, so they use the smaller DataAPIHandle:

const handle = await makeDataAPI(target);   // 'cloud' or 'local'
 
await handle.data.query({ query, variables });
const uid = await handle.getUid();
await handle.close();

It is the same DataAPI contract in a lighter wrapper. The factories (makeCloudDataAPI, makeLocalDataAPI) absorb the difference between stateless cloud calls and a stateful WebSocket connection.

Open and closed

Everything you need to use, extend, self-host or contribute is in the public repo:

• Contracts (smartchats-backend): the typed interface every consumer codes against.
• Local stack (smartchats-backend-local, smartchats-local-server, smartchats-database, smartchats-cli, smartchats-mcp): the install-and-run path. No cloud account needed.
• Cloud client (smartchats-cloud-client): the client for the hosted cloud. It defaults to smartchats.ai.

The code that runs the hosted service is closed. Everything in the public repository, every package included, is licensed under the Apache License 2.0 (copyright Sattvic Systems LLC).

Single direct importer of surrealdb

Only packages/smartchats-database/src/client.ts imports the surrealdb npm package in runtime code. Everything else goes through the Client interface it exports (createClient, createLazyClient, createUserClient, and the client’s signupAsUser and signinAsUser).

This made the v2 to v3 SDK migration a one-file change. SDK quirks (named exports, the connect() change, the removal of queryRaw, the Table wrapper for insert(), DateTime normalization) are handled inside client.ts.

Auth and credentials

There is one credential store, ~/.smartchats-mcp/credentials.json (mode 0600), which holds the cloud refresh token. The CLI, the MCP server, the bridge and the hub all use it. Run smartchats login once.

The ID token lasts one hour. smartchats-cloud-client refreshes it on demand with the stored refresh token over plain fetch. The browser sign-in flow uses a localhost callback.

Data model: event time and row lifecycle

Every event-time table has two kinds of timestamp. Keep them apart.

The event-time fields answer “when did it happen?”:

• ts: datetime: the UTC instant. Use it to sort and filter.
• local_date: string: YYYY-MM-DD in the user’s time zone at the time, so GROUP BY local_date gives correct daily totals with no time zone logic in the query.
• local_tz: string: the IANA zone (for example America/Chicago), so the local time of day can be rebuilt later.

Bundles import without created_at and updated_at, so the destination database stamps fresh values while the event-time fields keep the original timeline. Without this split, a year-old bundle would look “created today” and sort wrongly.

Schema change history is in HISTORY.md at the repo root.

Verification

The layered test runner is packages/smartchats-test. Each level runs in order and stops at the first failure unless you pass --continue-on-failure.

npx smartchats-test                 # every level that needs no running stack (L0-L3)
npx smartchats-test quick           # lint + build
npx smartchats-test build,unit      # chosen levels
npx smartchats-test --list          # list levels

For e2e, use bin/test-e2e. It starts the no-Docker stack with bin/test-bun-deploy, waits for /local-api/health, runs smartchats-test e2e, and shuts the stack down.

bin/test-e2e                        # full Simi suite
bin/test-e2e --pick                 # choose workflows interactively
bin/test-e2e -- --grep agent_input  # pass flags through to Playwright

See the Verification Workflow in CONTRIBUTING.md for which level fits which change.

See also

• Voice engine: vie, the turn judge and tuning.
• Providers: model and speech providers and keys.
• Local server: routes, SurrealDB and storage.
• Hub and Relay: agent messaging.
• MCP Server: the data tools for MCP clients.
• CLI Reference: every subcommand.
• Contributing: conventions and the dev workflow.