Proposal: Language Models and the Agent Runtime

How capOS runs language models — including a built-in on-ISO local model — as ordinary capability-served processes, and how the interactive agent is structured around an interactive tool-use loop instead of a plan-approve-execute pipeline.

Why This Proposal Exists

Two problems converge:

The earlier shell proposal sketched an “agent shell” that was itself a natural-language planner embedded in the shell process. That collapses three distinct concerns (user interaction, capability holding, model inference) into one, and it also got the shape of the interaction wrong: a one-shot “model emits a plan, user approves, dispatcher executes” pipeline is strictly weaker than how real agent systems work. In practice the model runs in a tool-use loop: it emits tool calls, the runtime executes them, results feed back into the conversation, the model decides what to do next, and the user stays in the loop through per-tool permission gates and interrupts. That interactive loop is what makes an agent useful; a static plan is a degenerate case of it.
capOS has no story for where model weights live, who holds them, what accelerator they run on, or how external model providers (remote HTTP, local Ollama, a future NPU) plug into the same interface. Every serious workload — interactive agent, chat NPCs in the adventure demo, summarisation of audit logs, semantic search over LogReader, embedding-based retrieval from a Directory — wants a language or embedding model. Without a shared capability surface, each consumer reinvents the wiring and smuggles different amounts of authority into the model process.

This proposal defines both halves: the model-as-capability architecture, and the agent-runner that drives the interactive tool-use loop on top of it.

Long-lived OpenClaw-like hosted agents, multi-agent swarms, workspace/memory control planes, MCP/A2A-style interoperability, and agent-maintained wiki substrates are split into capOS-Hosted Agent Swarms. This document keeps the base model and single-runner loop narrow.

Scope

Language models (chat / completion / tool use / structured output).
Text embedders (vector encoders for retrieval).
Tokenisers and small auxiliary models (classifier, reranker, guardrail).
A built-in local model shipped on the ISO for first-boot and offline use.
Pluggable external backends (remote HTTP providers, future GPU-accelerated local inference, future NPU).
The interactive agent runner that exposes session capabilities to the model as tools, executes tool calls, streams results back, and keeps the user in the loop.
A web-shell execution model where the browser agent is the UI and may orchestrate the LLM/tool-call loop, while WebShellGateway keeps capOS capabilities server-side and enforces every tool invocation.

Out of scope here (deferred):

Training, fine-tuning, RLHF pipelines. capOS is an inference host, not a trainer.
Native realtime multimodal voice sessions. The same authority split applies, but realtime audio, barge-in, transcripts, and provider tool-call events need a separate session interface; see Realtime Voice Agent Shell.
Long-lived hosted-agent swarms, external channel-triggered background work, durable task queues, agent-maintained wikis, MCP/A2A bridges, and OpenClaw-like harness control planes; see capOS-Hosted Agent Swarms.
Federated / multi-party inference. Treated as a later network topology.

Design Principles

Models are services, not shells. A model runs in a dedicated process with its own CapSet. It has no session cap, no TerminalSession, no Launcher, no ProcessSpawner, no ApprovalClient, no user secrets, and no inbound network authority. Its only job is to turn inputs into outputs through typed methods.
Prompts and outputs are data. Nothing the model reads or writes is authority by itself. The model cannot “say” a capability into existence. Free-form text it emits is never parsed as a command. Tool calls are a separate structured output channel — typed arguments, not shell lines.
Tool calls are proposals, not invocations. The model does not hold tool caps and does not perform the call. It emits a ToolCall value naming an advertised tool, with typed arguments conforming to the tool’s schema. A trusted capOS-side runner or WebShellGateway tool proxy decides whether to execute, prompt the user, or refuse.
Per-tool permission, not per-plan approval. Each tool carries a permission mode: auto (read-only, auto-execute), consent (ask the user quickly before running, similar to a per-action “Allow” prompt), stepUp (re-auth required), or forbidden (advertised for explanation only, never runnable). Permission lives on the tool descriptor, not on a post-hoc review of a generated plan. This matches how real agent systems behave and avoids the impossible review problem of a twenty-step plan.
The interface is the permission. A caller holding LanguageModel can request completions. A caller holding TextEmbedder can request vectors. Neither exposes weights, tokeniser internals, raw accelerator memory, or administration of the model service. Those stay behind separate ModelAdmin, ModelCatalog, and ModelRuntime caps held only by the service’s supervisor.
Backends are substitutable behind the same interface. LanguageModel does not imply on-host inference. A LanguageModel handle may be served by the built-in local model, an in-tree Rust inference engine, a GPU-accelerated local backend, or a wrapper over a remote provider. The caller cannot tell from the capability alone — and should not need to.
Weights are read-only file-backed memory. Weights live as files in the ISO (for the built-in model) or a storage volume (for installed models), and are mapped into the model process through a read-only file-backed MemoryObject. A shared page cache lets multiple model worker processes and multiple sessions share the same physical frames. Weights are never copied into process-private memory.
Policy lives in the broker, not in the model. AuthorityBroker decides which sessions get a LanguageModel cap, which backend the cap resolves to, which tools are advertised with which permission modes, rate and quota limits, and whether outbound network providers are allowed. The model enforces none of this; it cannot, because it does not see the session.
User interrupts beat model momentum. The user can break the loop at any time — Ctrl-C, a UI cancel, a terminal close. An in-flight tool call is either aborted or allowed to complete without its result going back to the model. The runner never waits for the model to “decide to stop”.
Browser agents are UI, not authority. In a web shell the agent may live in browser JavaScript and may call a provider API directly with an ephemeral token. That does not make it a capOS authority holder. Browser code can propose structured tool requests to WebShellGateway; the gateway and broker validate, authorize, execute, revoke, and audit.
Audit every tool call that touched authority. Each executed tool call is logged with model identity, model version, turn index, the advertised tool descriptors, the exact typed arguments, permission decision, user consent (if any), and the tool’s outcome. The model service does not write audit records; the runner does, because only the runner or gateway tool proxy sees both the call and the execution.

Architecture Overview

There are two accepted execution models. The native/capOS-side model keeps the whole agent loop inside a capOS process. The web-shell model lets the browser agent be the user interface and turn orchestrator, but not the holder of raw capOS capabilities.

CapOS-Side Runner

flowchart LR
    User[User / terminal] --> Runner[Agent Runner<br/>holds session caps]

    Runner -->|LanguageModel.complete| ModelSvc[language-model service process]
    ModelSvc --> Weights[(Read-only MemoryObject<br/>weights file)]
    ModelSvc --> Backend{Backend}
    Backend -->|cpu| CpuEngine[In-process inference engine]
    Backend -->|gpu| GpuSession[GpuSession cap]
    Backend -->|remote| Http[HttpEndpoint to provider]

    ModelSvc -. "text + tool calls" .-> Runner
    Runner -->|per-tool policy| Gate{Permission?}
    Gate -->|auto| Invoke[Invoke typed cap]
    Gate -->|consent| Prompt[Prompt user y/n]
    Prompt --> Invoke
    Gate -->|stepUp| Broker[AuthorityBroker step-up]
    Broker --> Invoke
    Gate -->|forbidden| Refuse[Refuse, feed error back]

    Invoke --> Services[Session caps: files, net, spawn, status...]
    Invoke --> Audit[AuditLog]
    Services -. "result" .-> Runner
    Runner -->|role:tool result| ModelSvc

Two principals matter in the capOS-side runner model:

Agent runner. Holds the session cap bundle (terminal, home, logs, launcher, approval, model client, etc.). Runs the user-facing loop, talks to the model, applies per-tool permission policy, executes tool calls against its held caps, streams results back to the model, and writes audit. This is the natural daily driver — either the native shell in “agent mode” or a sibling process launched from the shell.
Model service. Holds weights, an optional accelerator session, and an optional narrow outbound HttpEndpoint for remote backends. Sees conversation messages; emits text and tool calls. Has no session, no tools, no spawn authority.

The kernel does not need a “model” or “agent” concept. Everything here is ordinary capabilities, processes, and ring traffic.

Browser Agent UI

In a web shell, the agent itself may be the UI. Browser JavaScript may render the conversation, call a provider LLM API directly, receive structured tool calls, and feed tool results back into the model. That mode exists for latency, provider-native browser SDKs, and richer UI composition.

It still does not give browser JavaScript raw capOS capabilities:

flowchart LR
    User[User] --> BrowserAgent[Browser Agent UI<br/>LLM loop]
    BrowserAgent -->|ephemeral provider token| Provider[LLM Provider API]
    Provider -. "text + tool calls" .-> BrowserAgent

    BrowserAgent -->|ToolRequest| Gateway[WebShellGateway<br/>ToolProxy]
    Gateway --> Broker[AuthorityBroker]
    Gateway --> Audit[AuditLog]
    Gateway --> Services[Session caps: files, net, spawn, status...]
    Services -. "typed result" .-> Gateway
    Gateway -. "ToolResult" .-> BrowserAgent
    BrowserAgent -. "tool result" .-> Provider

Authority split:

Browser agent UI. Owns presentation, local conversation state, user gestures, optional browser media APIs, and direct provider session state. It holds no capOS caps, no session caps, no tool caps, and no provider long-lived credentials.
WebShellGateway tool proxy. Owns the authenticated web transport and the server-side reference to the session bundle. It exposes the current tool descriptor snapshot to the browser, accepts structured ToolRequest values, validates them against the session, enforces broker policy and consent/step-up, invokes the real capOS capabilities, and writes audit.
Provider. Sees prompts and tool results only when broker policy allows direct browser provider use for the session’s confidentiality profile.

The browser-agent model is therefore browser-orchestrated but gateway-enforced. It is not a bearer-capability model and not a shortcut around the broker.

The Tool-Use Loop

One capOS-side agent turn:

1. User types a message (or kicks off the first turn from a CLI arg).
2. Runner assembles: system prompt + prior messages + user message +
   the set of ToolDescriptor values the session currently advertises.
3. Runner calls LanguageModel.stream(req). Token stream is rendered to
   the terminal as it arrives.
4. Model response finishes. It contains text (shown) plus zero or more
   ToolCall records (not shown as text; shown as typed tool-call UI).
5. For each ToolCall:
     a. Look up the tool by name. If not in the advertised set, reject
        with a typed error fed back as a role: tool result.
     b. Validate arguments against the tool's paramSchema. Reject if
        malformed; feed the validation error back.
     c. Check the tool's permission mode:
          - auto:      proceed.
          - consent:   render the call + arguments + permission UI;
                       wait for user y/n. Deny feeds a refusal back.
          - stepUp:    request a leased narrow cap from the broker,
                       possibly driving WebAuthn/OIDC step-up. On
                       success, proceed; on denial, feed back.
          - forbidden: reject; feed typed "not permitted in this
                       session" error back.
     d. Invoke the underlying typed capability. Time-box the call.
     e. Truncate/redact the result per tool policy, serialize as a
        role: tool message keyed to the ToolCall id.
6. If the model emitted tool calls, loop back to step 3 with the
   results appended. If it emitted none (or the user interrupted),
   this turn ends.
7. Every executed call produces an audit record.

One browser-agent UI turn:

1. User interacts with the browser agent UI.
2. Browser agent assembles the prompt, prior messages, and the current
   ToolDescriptor snapshot fetched from WebShellGateway.
3. Browser agent calls the provider directly using a broker-minted,
   short-lived, provider-scoped token.
4. Provider response streams into the browser. If it emits ToolCall records,
   the browser wraps each as a ToolRequest to WebShellGateway.
5. WebShellGateway validates the call against the advertised descriptor,
   current session state, nonce/turn binding, quotas, and broker policy.
6. WebShellGateway obtains any required consent or step-up proof, invokes the
   underlying capOS capability server-side, writes audit, and returns a
   ToolResult.
7. Browser agent feeds the ToolResult back to the provider and continues the
   loop until no tool calls remain or the user/gateway cancels.

Browser-originated tool requests are untrusted input even when the agent is the intended UI. The gateway must reject stale descriptors, unknown tools, argument/schema mismatches, replayed turn ids, requests outside the current session profile, and any operation whose consent or step-up proof is missing.

Interactive-agent niceties that fall out of this structure:

Streaming. Tokens render live. Tool calls appear as structured widgets, not as text the user has to parse.
Interruption. Ctrl-C at any point cancels the in-flight inference (TokenStream.cancel) or the in-flight tool call. The runner decides whether to feed a cancellation message back to the model or end the turn.
Auto vs. consent. Reading files, listing directories, querying SystemStatus, reading logs — auto. Writing files, spawning processes, changing service state, sending network requests — consent. Destroying data, running a recovery operation, widening the session’s own caps — stepUp or forbidden.
Context management. When the transcript approaches ModelInfo.contextTokens, the runner can summarise older turns (via a second LanguageModel.complete call) and replace them with a compact summary message. This is a runner decision, not a kernel or model feature.
Conversation persistence. A conversation is a list of messages plus a reference to the runner’s session; it can be written to a home-scoped file, resumed later, forked, or compared. Persistence is an ordinary capability concern, handled by the runner through whatever Directory/File cap it holds.

Agent Mode is a Mode of the Native Shell

The native shell from shell-proposal.md becomes the agent runner. It already holds the session bundle; adding a LanguageModel client cap plus a per-tool permission table gives it “agent mode”. In that mode:

Plain user input becomes a chat turn.
/cap, /inspect, /exit, and the other existing direct commands stay as direct typed invocations that bypass the model.
Tool descriptors are generated by the same schema reflection the shell already needs for its capability REPL.

A separate capos-agent binary is possible for deployments where agent mode is the default (think “bare capOS image with no traditional shell”). It launches from the same login path with the same session bundle, differs only in the surface presented to the user.

Web Agent Mode is a Mode of WebShellGateway

For browser-hosted sessions, WebShellGateway exposes an agent UI protocol instead of making the browser a capOS process. The protocol can be JSON over WebSocket or another web-native framing, but its values mirror ToolDescriptor, ToolCall, and ToolResult from this proposal.

Gateway responsibilities:

issue short-lived provider credentials only when policy allows direct browser LLM access;
bind the tool descriptor snapshot to a session id, conversation id, turn id, expiration, and browser connection;
execute tools only through server-side session caps;
enforce low-risk consent, mutating consent, and destructive stepUp server-side;
return redacted/truncated tool results according to tool policy;
revoke or expire provider tokens where the provider supports it, reject new tool requests on logout, timeout, tab close, session downgrade, or policy change, and record any browser-held provider session that can only be terminated best-effort.

Browser responsibilities:

render the agent UI and any consent prompts supplied by the gateway;
preserve provider session state only as long as the gateway session is live;
submit structured tool requests, never raw capability invocations;
treat gateway denials, cancellation, and revocation as authoritative.

For mutating or destructive tools, a browser click is not enough by itself. The gateway needs a fresh server-side consent challenge or a broker-issued step-up lease tied to the exact tool name, arguments, conversation, turn, and expiration. Low-risk read-only tools may use auto execution when broker policy allows.

Prompt Injection as a First-Class Concern

Model inputs include untrusted data: file contents, log lines, web pages fetched via a tool call, Aurelian Frontier NPC dialogue, output from previously executed tool calls. Every such input is wrapped in a role: user or role: tool message with explicit provenance, never concatenated into a system prompt. The runner never parses assistant free text as a command, and the gateway never treats browser-submitted free text as a capability request. Only structured toolCalls / ToolRequest values can reach the tool execution path.

A user can paste rm -rf / at the model; the model can repeat it back; nothing happens, because there is no code path that interprets text as a command. A web page can instruct the model to exfiltrate secrets; the model cannot use capOS resources except through the advertised tool set, and sensitive tools are gated by consent/stepUp. If the browser agent has ordinary web-network reachability, broker policy must treat prompts and tool results as exposed to that browser/provider boundary and deny direct browser mode for sessions where that is unacceptable.

Capability Contract

Additions to schema/capos.capnp (exact method IDs and argument packing belong to the implementation PR; the shapes below are the contract):

# Conversation inputs/outputs are plain data. They carry no authority.

struct ChatMessage {
  role @0 :Role;
  content @1 :Text;
  # For role:tool, the id of the ToolCall this message answers.
  toolCallId @2 :Text;
  # For role:assistant messages that included tool calls, the list
  # of calls the model proposed.
  toolCalls @3 :List(ToolCall);
}

enum Role {
  system @0;
  user @1;
  assistant @2;
  tool @3;
}

struct ToolDescriptor {
  name @0 :Text;
  description @1 :Text;
  # Capability and method the runner will invoke if this tool fires.
  interfaceId @2 :UInt64;
  methodName @3 :Text;
  # JSON-Schema or equivalent describing the argument object.
  paramSchema @4 :Text;
  # Permission mode. Enforced by the runner, surfaced to the model as
  # hint metadata so the model can explain or avoid risky calls.
  permission @5 :PermissionMode;
  # Tool category for audit and policy filters.
  category @6 :Text;
}

enum PermissionMode {
  auto @0;        # Runner executes without user prompt.
  consent @1;     # Runner prompts user before execution.
  stepUp @2;      # Runner requests broker step-up before execution.
  forbidden @3;   # Advertised for explanation only; never executed.
}

struct ToolCall {
  id @0 :Text;          # Unique within the conversation.
  name @1 :Text;
  # Arguments serialised as JSON (or capnp AnyPointer in a later
  # revision). The runner validates against paramSchema.
  arguments @2 :Text;
}

struct ToolResult {
  callId @0 :Text;
  outcome @1 :Outcome;
  content @2 :Text;     # Possibly truncated / redacted by the runner.
  error @3 :Text;       # Set when outcome != ok.
}

enum Outcome {
  ok @0;
  refusedByPolicy @1;
  deniedByUser @2;
  stepUpFailed @3;
  executionError @4;
  timedOut @5;
  cancelled @6;
  invalidArguments @7;
  unknownTool @8;
}

struct InferenceRequest {
  messages @0 :List(ChatMessage);
  tools @1 :List(ToolDescriptor);
  maxTokens @2 :UInt32;
  temperature @3 :Float32;
  stopSequences @4 :List(Text);
  # Optional JSON-Schema for final-assistant structured output.
  responseSchema @5 :Text;
  # Stable correlation id for audit.
  nonce @6 :Data;
}

struct InferenceResponse {
  message @0 :ChatMessage;   # role:assistant, may include toolCalls.
  usage @1 :TokenUsage;
  finishReason @2 :FinishReason;
}

interface LanguageModel {
  info @0 () -> (info :ModelInfo);
  complete @1 (req :InferenceRequest) -> (resp :InferenceResponse);
  # Streaming variant emits token chunks and tool-call deltas as they
  # are decoded. Cancellation aborts decoding.
  stream @2 (req :InferenceRequest) -> (stream :TokenStream);
}

interface TokenStream {
  next @0 () -> (chunk :StreamChunk, done :Bool);
  cancel @1 () -> ();
}

struct StreamChunk {
  textDelta @0 :Text;
  toolCallDelta @1 :ToolCallDelta;   # partial structured tool call
}

interface TextEmbedder {
  info @0 () -> (info :ModelInfo);
  embed @1 (texts :List(Text)) -> (vectors :List(Vector));
}

struct ModelInfo {
  id @0 :Text;           # Content-addressed weight digest + arch tag.
  displayName @1 :Text;
  arch @2 :Text;         # "llama", "qwen", "phi", etc.
  contextTokens @3 :UInt32;
  outputTokens @4 :UInt32;
  backend @5 :Text;      # "local-cpu", "local-gpu", "remote-openai", ...
  quantisation @6 :Text; # "fp16", "q4_k_m", ...
  supportsTools @7 :Bool;
}

# Administrative surface. Not granted to normal sessions.
interface ModelCatalog {
  list @0 () -> (models :List(ModelInfo));
  openLanguageModel @1 (id :Text) -> (model :LanguageModel);
  openEmbedder @2 (id :Text) -> (embedder :TextEmbedder);
}

interface ModelAdmin {
  loadWeights @0 (source :ReadOnlyFile, info :ModelInfo) -> (id :Text);
  unload @1 (id :Text) -> ();
  setBackendPolicy @2 (policy :BackendPolicy) -> ();
}

The web-shell protocol should expose a non-capability tool proxy with the same data shapes. Exact framing belongs to the WebShellGateway milestone:

describeTools(session, conversation) -> List(ToolDescriptor)
invokeTool(session, conversation, turn, descriptorSnapshot, ToolCall)
    -> ToolResult
cancelTurn(session, conversation, turn) -> ()

This is intentionally not a LanguageModel method and not a capOS capability handle passed to the browser. It is an authenticated web transport endpoint whose implementation invokes real session caps only after gateway/broker checks pass.

What is deliberately absent

No method on LanguageModel accepts a capability argument. The model never holds a live cap to a user resource.
No method returns a capability that could be invoked outside the model service (TokenStream is the one exception and is scoped to the current response).
No “run this tool for me” method on LanguageModel or any model service. Tool execution is the runner’s or gateway tool proxy’s job. The model only names tools.
No PlannerAgent / ActionPlan / dispatcher interface. Planning, if it happens, is something a model does inside one of its responses; it is not a separate typed product.
No “agent shell interface” served by the model. In the capOS-side model, the shell is the runner and capability holder; in the browser-agent model, WebShellGateway is the capability holder.

The Agent Runner

This section describes the capOS-side runner. Browser-hosted sessions use the WebShellGateway tool proxy described above instead of placing the runner and session caps in browser JavaScript.

The runner is an ordinary userspace process (native shell in agent mode, or capos-agent) that holds:

The session cap bundle, unchanged from the shell proposal.
A LanguageModel client cap issued by the broker.
A ModelInfo read-only view for rendering model identity.
A ConversationStore cap (when one exists) for persistence.

It does not hold ModelCatalog or ModelAdmin — those are administrative. If a session wants to switch models mid-run, the broker issues a new LanguageModel cap.

Building the Tool Table

On startup (and after any cap-set change), the runner walks its own session bundle and produces ToolDescriptor values through schema reflection over the advertised capabilities’ interfaces. It applies the broker-supplied per-tool permission map keyed by (category, methodName):

read-only       -> auto
mutating local  -> consent
destructive     -> stepUp
outbound net    -> consent (unless profile allows auto)
admin-class     -> forbidden (for non-operator sessions)

The runner is free to suppress tools entirely for a given conversation (for example, never advertise ServiceSupervisor.restart for a guest session, even though the descriptor set could carry a forbidden entry). Suppression is sometimes clearer than presenting an unusable tool to the model.

The Loop State Machine

Idle
  │  user turn arrives
  ▼
AssemblingRequest ── tool-descriptor snapshot ─► Inferring (LanguageModel.stream)
  ▲                                                     │
  │ tool result appended                                │ model finishes
  │                                                     ▼
ExecutingCalls ◄─── one call at a time ───────── HasToolCalls?
     │ per call: gate → execute → audit               │ no
     └──────────────────────────────────────────┐     ▼
                                                │    Idle
                                                ▼
                                    (any denial / cancel is
                                     an outcome fed back)

Timeouts are enforced at three levels: per-tool (so a slow capability does not block the loop forever), per-turn (bounded number of iterations to prevent runaway), and per-session (token and wall-clock budgets from the broker).

Conversation State

A conversation is List(ChatMessage) plus a ModelInfo.id, the effective ToolDescriptor table at each turn, and the audit trail. The runner keeps it in its own process memory during a session and may persist it through a ConversationStore cap (when that exists; see open questions). No conversation state lives in the model service; the service is stateless across requests.

The Built-in Local Model

capOS ships with a small local language model so that:

First boot has a working agent without remote network.
The adventure and chat demos can have a real local NPC brain rather than hard-coded strings.
Offline and air-gapped deployments remain viable.
The capability surface has a real local implementation to validate against before remote backends are wired up.

Constraints

Size budget. A 1–3 B parameter quantised model (q4_k_m-class) fits in 0.7–2.0 GiB. That is too large for manifest.bin embedding (2.75 MiB cap) and forces the ISO filesystem path — see the Boot Binary ISO Layout item in docs/backlog/hardware-boot-storage.md. Weights are the first non-binary consumer of the ISO file path.
Tool calling. The model must be a tool-use-capable instruction tune (a chat-tuned model without reliable tool-call formatting cannot drive the loop). ModelInfo.supportsTools flags this.
Backend. First implementation is CPU-only, portable Rust inference. Candidates include candle (needs no_std survey), a minimal hand-rolled GGUF loader + matmul kernel, or a vendored subset of a permissively licensed engine. Final choice is an implementation decision, not a proposal decision; the capability surface is implementation-agnostic.
Precision. q4_k_m or q5_k_m quantised GGUF. fp16 is a later optimisation gated on either SIMD-friendly CPU support or GPU acceleration.
Context window. 4 K–8 K tokens at first. Enough for short agent sessions; long-document summarisation is a later workload that may require a different model or aggressive runner-side compaction.
Attestation. Weights are signed (see cryptography-and-key-management-proposal.md) and the signature is verified at load. The content-addressed digest becomes the ModelInfo.id.

Boot Flow

ISO driver (pending the Boot Binary ISO Layout item in docs/backlog/hardware-boot-storage.md) exposes /boot/models/<name>.gguf as an ordinary file.
Kernel or a privileged loader service constructs a read-only file-backed MemoryObject over the weights file. Read-only shared frames let multiple model worker processes map the same weights without copies.
model-loader service (started from the manifest) verifies the signature, registers the model in ModelCatalog, and keeps a retained handle to the weights MemoryObject.
On demand, ModelCatalog.openLanguageModel(id) spawns (or returns a handle to) a worker process holding the weights, an inference kernel, and — if policy allows — a GpuSession or a remote HttpEndpoint.

Weights never live in the manifest blob. The ISO layout work is the prerequisite, and this proposal is its first forcing use case larger than a few megabytes.

Page Cache Coupling

Multiple sessions sharing one model benefit from a page cache over the weights file: the first access faults in, subsequent accesses hit cache, and the pages are shared read-only across all worker processes. This is the same primitive that makes ELF text-segment sharing useful, and it should be implemented once in the ISO/file-backed-memory path rather than specialised per consumer.

CapOS-Side Backends

CapOS-side backends sit behind LanguageModel / TextEmbedder. The worker process loads exactly one backend per instance. Browser direct-provider mode is a separate web transport mode described below; it is not a LanguageModel worker backend.

Local CPU

File-backed read-only weights mapped from ISO or storage.
No accelerator caps. No network caps.
Bounded per-call token budget enforced by the worker; broker sets per-cap quotas.

Local GPU

Holds a GpuSession from the GPU capability proposal.
Holds a read-only MemoryObject for the weights; uploads to GPU memory at load time through GpuBuffer.
Still no network. Still no session cap.

Remote Provider

Holds one narrow HttpEndpoint scoped to a single provider origin (for example an Ollama instance on the local network, or an external API gateway). The endpoint is issued by the broker; the model worker cannot widen it.
Holds provider credentials only as token-typed capabilities (OAuth AccessToken wrapped as a cap, never exposed as a bearer string — see OIDC and OAuth2 proposal).
The model worker process is still the principal that talks to the remote; the runner never sees provider credentials.
Treated as untrusted: outbound request/response logging is mandatory when operator policy requires audit of off-device inference.

NPU / Future Accelerators

Same shape. Add a scoped NpuSession cap analogous to GpuSession when the hardware abstraction for it exists.

Browser Direct-Provider Mode

Browser receives only a broker-minted ephemeral credential scoped to one provider, model/config, session, conversation, and short expiration.
The credential contains no capOS capability material and cannot be exchanged for session caps.
The browser may run the provider’s JavaScript/WebRTC/WebSocket client and orchestrate the LLM loop.
Tool execution still goes through WebShellGateway’s tool proxy; provider tool declarations must match the gateway-advertised descriptors for that turn.
Broker policy may deny this mode for sessions whose prompts, tool results, labels, or audit requirements cannot leave the capOS-side trust boundary.
Logout, tab close, timeout, or session downgrade authoritatively closes the capOS session and rejects future tool requests. Provider token/session revocation is authoritative only when the provider exposes a server-side revocation or session-close API; otherwise it is best-effort and must be audited as such.

Policy and the Broker

AuthorityBroker gates every model interaction:

Which session profiles get a LanguageModel cap at all (operator: yes; anonymous: usually no; guest: local-only, no remote providers).
Which backend resolves an openLanguageModel(id) call for this session (local-only for unclassified work; remote permitted for operators who opted in and passed step-up auth).
Rate and token-budget limits per session and per principal.
The per-tool permission map the runner applies when building its tool table, or that WebShellGateway applies before publishing descriptor snapshots to a browser agent. This is the main policy knob: an anonymous session might get only read-only tools as auto; an operator session gets consent on mutating tools and stepUp on destructive ones.
Outbound-network egress policy for remote backends.
Whether direct browser provider access is allowed for this session, and which prompts, transcripts, tool descriptors, and tool results may cross that browser/provider boundary.
PII / confidentiality labels: a session labelled MAC/MIC-high may be denied remote inference entirely because prompts would cross the confidentiality boundary (see formal-mac-mic-proposal.md).

The broker’s decisions are recorded in audit. The model service itself performs no policy checks — it is an execution backend.

Audit and Provenance

Every executed tool call audit record includes:

Session ID, principal, conversation ID, turn index, tool-call ID.
Model identity (ModelInfo.id), backend, request nonce.
Runner location (capos-side or browser-agent-ui) and gateway session id when a browser agent proposed the call.
Advertised tool descriptor at the time of the call (name, paramSchema, permission mode).
Exact typed arguments.
Permission decision (auto, consented, denied, step-up-succeeded, step-up-failed, forbidden).
Tool outcome, truncated result hash, and error if any.

Optional per-session conversation-level records capture message metadata (role, timestamp, length, hash) without requiring full prompt content to be stored — the classification policy decides how much content is retained.

This lets an operator answer “what did the agent do on my behalf last week, which model produced each call, and which tools were visible” without replaying prompts from logs the model service does not hold.

Threat Model

Assumed hostile:

Prompts, retrieved documents, web pages, and tool-call outputs.
Model weights from unknown sources (mitigated by weight signing and ModelInfo.id attestation).
The model worker process itself — treated as a semi-trusted data transformer, isolated by its narrow CapSet.
Browser JavaScript, browser extensions, DOM state, browser-held provider sessions, and browser-agent UI code. They may be the intended user interface, but their tool requests are untrusted inputs to the gateway.

Assumed trustworthy (with attestation):

The kernel, the capOS-side runner when used, WebShellGateway’s server-side tool proxy, the broker, the ISO driver, and the loader.

Out of scope (covered by other proposals or tracks):

Side-channel leakage through cache timing on shared accelerators — follow work on GPU tenant isolation in the GPU proposal.
Model-backdoor detection — an ecosystem problem, not a kernel one; capOS only guarantees that a compromised weights file cannot escape its worker process’s CapSet.

Integration with Existing Workloads

Adventure demo. NPC processes can hold a narrow LanguageModel cap scoped to small prompt budgets, producing in-character lines instead of canned strings. Chat rooms can feed the demo through a runner variant without session-level tools.
Boot-to-shell first-use. First boot can offer an agent-assisted setup flow (“help me configure the network stack”) once the runner is wired up and the operator session profile includes the model cap and the right tool permission map.
Log and metric summarisation. LogReader becomes a consent-gated tool in the runner’s tool table. The model asks for “last hour of auth errors”; the runner executes, truncates, feeds back. The model never holds LogReader itself.
Semantic search over directories. TextEmbedder + a vector index service (future) lets home/docs-scoped search work through a search tool advertised by the runner, without ambient file access for the model.

Implementation Phases

Phase 0 — Prerequisites

ISO 9660 driver + file-backed read-only MemoryObject (docs/backlog/hardware-boot-storage.md and the follow-on file-backed memory work).
Page cache over file-backed memory.
HttpEndpoint scoped-origin fetch (networking proposal Phase B).
AuthorityBroker and ApprovalClient wiring (shell proposal implementation plan step 4).
Schema reflection sufficient to build ToolDescriptor values.

Phase 1 — Capability scaffolding

Add LanguageModel, TextEmbedder, ModelInfo, ModelCatalog, ModelAdmin, ToolDescriptor, PermissionMode, ToolCall, ToolResult, InferenceRequest, InferenceResponse, StreamChunk, TokenStream to schema/capos.capnp.
Generate bindings via existing tools/capnp-build.
Stub language-model service process with a deterministic canned-tool-call backend so the runner loop can be exercised without any real inference.
make run-agent smoke: shell in agent mode runs a scripted conversation through the stub, exercises auto / consent / stepUp / forbidden gates, and exits cleanly.

Phase 2 — Built-in local model

Choose a CPU inference engine and vendor it.
Ship one tool-use-capable quantised model in iso_root/boot/models/ as a content-addressed GGUF with a signature.
Loader service verifies signature, maps weights, registers in ModelCatalog.
First real tool-use loop with a local model.

Phase 3 — Runner features

Streaming render into TerminalSession with interrupt support.
Context-budget compaction (summarise older turns via a secondary inference call).
Per-tool consent UI.
Audit integration.
Conversation persistence through a ConversationStore cap.
WebShellGateway tool proxy: descriptor snapshots, turn binding, replay rejection, server-side consent/step-up enforcement, and browser-agent-proposed audit records.

Phase 4 — Backends

GPU backend wired through GpuSession.
Remote-provider backend wired through HttpEndpoint + token-typed capability. One concrete provider (for example local Ollama) as the proof.
Broker policies for backend selection.
Browser direct-provider mode: broker-minted ephemeral credentials, short token expiry, provider revocation/close when supported, audited best-effort teardown otherwise, and a web-agent smoke that proves browser-orchestrated tool calls are executed only through WebShellGateway.

Phase 5 — Hardening and features

Structured-output (JSON/capnp) validation against responseSchema.
Embedding-backed retrieval service (TextEmbedder + vector store).
Prompt redaction for MAC/MIC-high sessions.
Audit replay tooling.
Step-up integration with the broker’s WebAuthn/OIDC paths.

Phase 6 — Applications

Agent-assisted adventure NPCs with per-NPC caps.
Agent-assisted first-boot setup flow.
Log-summarisation and monitoring assistant.
Optional: agent mode over the POSIX compatibility layer, once that exists.

Dependencies

Hard prerequisites:

ISO filesystem driver and file-backed MemoryObject (docs/backlog/hardware-boot-storage.md plus file-backed memory follow-on).
AuthorityBroker, ApprovalClient (shell proposal plan step 4).
WebShellGateway authenticated transport and server-side session tracking.
ProcessSpawner with exact-grant child launch (done).
Schema reflection / SchemaRegistry.
Cap’n Proto schema evolution tooling (done).

Soft / enables richer behaviour:

GPU capability proposal for GPU backend.
OIDC/OAuth2 proposal for remote-provider credentials and step-up authentication.
WebAuthn/passkey support for browser step-up on destructive tools.
Cryptography/KMS proposal for weight signing.
System monitoring proposal for audit integration.
Formal MAC/MIC proposal for high-confidentiality session policy.

Non-Goals

No kernel-side model awareness.
No ambient “AI” privilege anywhere.
No model-issued capabilities.
No long-lived bearer-token exposure to the runner or browser. Browser-agent UI mode may use only short-lived provider-scoped credentials.
No promise that any particular model size, license, or benchmark score ships in-tree — the choice is an implementation decision gated by the trusted-build-inputs process.
No plan/approve/execute pipeline as the primary interaction (explicitly superseded by the tool-use loop).
No claim that capOS offers strong defences against model-internal adversarial attacks (jailbreaks, refusal bypass). The capability model defends the system, not the model’s own behaviour.

Open Questions

Should tool arguments be JSON (matches provider ABIs like OpenAI tools / Anthropic tools) or capnp AnyPointer (matches capOS wire format)? Proposed: start with JSON for compatibility with remote providers and because local GGUF tool-use tunes are JSON-trained, and add a capnp fast path later.
How are conversations named, persisted, and resumed? A ConversationStore cap with TTL is the sketch, but the storage proposal needs an update before this is concrete.
What is the smallest credible local model that still drives the tool-use loop reliably for capOS-internal tasks (file edits, status summaries, NPC dialogue)? Below a threshold, better to ship no default model and require explicit configuration.
How should streaming back-pressure compose with ring cap_enter completion limits? A single response can produce many small CQEs.
When consent prompts pile up in a long turn, how should the runner offer “approve-once” vs. “approve-for-this-turn” vs. “approve-for-this-session” without widening authority beyond what the user intended? A per-session “always allow this tool” allow-list, cleared at session end, is a reasonable starting point.
Should the runner ever let the model read tool descriptors for tools it cannot execute (forbidden), so the model can explain why it can’t help, or should those be suppressed entirely?
Does the built-in model warrant its own trust anchor in the weights signing chain, or should it share the system trust store? Likely share, with a dedicated key purpose (see cryptography proposal).
Which web-shell profiles should allow browser-agent UI mode by default? Operator sessions may want it for latency and provider UX; high-label or audit-strict sessions should probably force capOS-side provider mediation.
How should the gateway prove fresh user presence for browser-agent approvals without trusting arbitrary JavaScript events? WebAuthn/passkey step-up handles destructive tools; low-risk consent still needs a concrete freshness rule.

Keyboard shortcuts

capOS Documentation