# Proposal: Language Models and the Agent Runtime
How capOS runs language models — including a built-in on-ISO local model — as
ordinary capability-served processes, and how the interactive agent is
structured around an interactive tool-use loop instead of a
plan-approve-execute pipeline.
## Why This Proposal Exists
Two problems converge:
1. The earlier [shell proposal](shell-proposal.md) sketched an "agent shell"
that was itself a natural-language planner embedded in the shell
process. That collapses three distinct concerns (user interaction,
capability holding, model inference) into one, and it also got the
*shape* of the interaction wrong: a one-shot "model emits a plan,
user approves, dispatcher executes" pipeline is strictly weaker than
how real agent systems work. In practice the model runs in a
tool-use loop: it emits tool calls, the runtime executes them,
results feed back into the conversation, the model decides what to
do next, and the user stays in the loop through per-tool permission
gates and interrupts. That interactive loop is what makes an agent
useful; a static plan is a degenerate case of it.
2. capOS has no story for where model weights live, who holds them, what
accelerator they run on, or how external model providers (remote HTTP,
local Ollama, a future NPU) plug into the same interface. Every
serious workload — interactive agent, chat NPCs in the adventure
demo, summarisation of audit logs, semantic search over `LogReader`,
embedding-based retrieval from a `Directory` — wants a language or
embedding model. Without a shared capability surface, each consumer
reinvents the wiring and smuggles different amounts of authority into
the model process.
This proposal defines both halves: the model-as-capability architecture,
and the agent-runner that drives the interactive tool-use loop on top of
it.
Long-lived OpenClaw-like hosted agents, multi-agent swarms, workspace/memory
control planes, MCP/A2A-style interoperability, and agent-maintained wiki
substrates are split into
[capOS-Hosted Agent Swarms](hosted-agent-swarm-proposal.md). This document
keeps the base model and single-runner loop narrow.
## Scope
- Language models (chat / completion / tool use / structured output).
- Text embedders (vector encoders for retrieval).
- Tokenisers and small auxiliary models (classifier, reranker, guardrail).
- A built-in local model shipped on the ISO for first-boot and offline use.
- Pluggable external backends (remote HTTP providers, future GPU-accelerated
local inference, future NPU).
- The interactive agent runner that exposes session capabilities to the
model as tools, executes tool calls, streams results back, and keeps the
user in the loop.
- A web-shell execution model where the browser agent is the UI and may
orchestrate the LLM/tool-call loop, while `WebShellGateway` keeps capOS
capabilities server-side and enforces every tool invocation.
Out of scope here (deferred):
- Training, fine-tuning, RLHF pipelines. capOS is an inference host, not a
trainer.
- Native realtime multimodal voice sessions. The same authority split applies,
but realtime audio, barge-in, transcripts, and provider tool-call events need
a separate session interface; see
[Realtime Voice Agent Shell](realtime-voice-agent-shell-proposal.md).
- Long-lived hosted-agent swarms, external channel-triggered background work,
durable task queues, agent-maintained wikis, MCP/A2A bridges, and OpenClaw-like
harness control planes; see
[capOS-Hosted Agent Swarms](hosted-agent-swarm-proposal.md).
- Federated / multi-party inference. Treated as a later network topology.
## Design Principles
1. **Models are services, not shells.** A model runs in a dedicated process
with its own `CapSet`. It has no session cap, no `TerminalSession`, no
`Launcher`, no `ProcessSpawner`, no `ApprovalClient`, no user secrets,
and no inbound network authority. Its only job is to turn inputs into
outputs through typed methods.
2. **Prompts and outputs are data.** Nothing the model reads or writes is
authority by itself. The model cannot "say" a capability into
existence. Free-form text it emits is never parsed as a command. Tool
calls are a separate structured output channel — typed arguments, not
shell lines.
3. **Tool calls are proposals, not invocations.** The model does not hold
tool caps and does not perform the call. It emits a `ToolCall` value
naming an advertised tool, with typed arguments conforming to the tool's
schema. A trusted capOS-side runner or `WebShellGateway` tool proxy decides
whether to execute, prompt the user, or refuse.
4. **Per-tool permission, not per-plan approval.** Each tool carries a
permission mode: `auto` (read-only, auto-execute), `consent` (ask the
user quickly before running, similar to a per-action "Allow" prompt),
`stepUp` (re-auth required), or `forbidden` (advertised for
explanation only, never runnable). Permission lives on the tool
descriptor, not on a post-hoc review of a generated plan. This
matches how real agent systems behave and avoids the impossible
review problem of a twenty-step plan.
5. **The interface is the permission.** A caller holding `LanguageModel`
can request completions. A caller holding `TextEmbedder` can request
vectors. Neither exposes weights, tokeniser internals, raw
accelerator memory, or administration of the model service. Those
stay behind separate `ModelAdmin`, `ModelCatalog`, and `ModelRuntime`
caps held only by the service's supervisor.
6. **Backends are substitutable behind the same interface.**
`LanguageModel` does not imply on-host inference. A `LanguageModel`
handle may be served by the built-in local model, an in-tree Rust
inference engine, a GPU-accelerated local backend, or a wrapper over
a remote provider. The caller cannot tell from the capability alone —
and should not need to.
7. **Weights are read-only file-backed memory.** Weights live as files in
the ISO (for the built-in model) or a storage volume (for installed
models), and are mapped into the model process through a read-only
file-backed `MemoryObject`. A shared page cache lets multiple model
worker processes and multiple sessions share the same physical
frames. Weights are never copied into process-private memory.
8. **Policy lives in the broker, not in the model.** `AuthorityBroker`
decides which sessions get a `LanguageModel` cap, which backend the
cap resolves to, which tools are advertised with which permission
modes, rate and quota limits, and whether outbound network providers
are allowed. The model enforces none of this; it cannot, because it
does not see the session.
9. **User interrupts beat model momentum.** The user can break the loop
at any time — Ctrl-C, a UI cancel, a terminal close. An in-flight
tool call is either aborted or allowed to complete without its result
going back to the model. The runner never waits for the model to
"decide to stop".
10. **Browser agents are UI, not authority.** In a web shell the agent may
live in browser JavaScript and may call a provider API directly with an
ephemeral token. That does not make it a capOS authority holder. Browser
code can propose structured tool requests to `WebShellGateway`; the
gateway and broker validate, authorize, execute, revoke, and audit.
11. **Audit every tool call that touched authority.** Each executed tool
call is logged with model identity, model version, turn index, the
advertised tool descriptors, the exact typed arguments, permission
decision, user consent (if any), and the tool's outcome. The model
service does not write audit records; the runner does, because only
the runner or gateway tool proxy sees both the call and the execution.
## Architecture Overview
There are two accepted execution models. The native/capOS-side model keeps
the whole agent loop inside a capOS process. The web-shell model lets the
browser agent be the user interface and turn orchestrator, but not the holder
of raw capOS capabilities.
### CapOS-Side Runner
```mermaid
flowchart LR
User[User / terminal] --> Runner[Agent Runner
holds session caps]
Runner -->|LanguageModel.complete| ModelSvc[language-model service process]
ModelSvc --> Weights[(Read-only MemoryObject
weights file)]
ModelSvc --> Backend{Backend}
Backend -->|cpu| CpuEngine[In-process inference engine]
Backend -->|gpu| GpuSession[GpuSession cap]
Backend -->|remote| Http[HttpEndpoint to provider]
ModelSvc -. "text + tool calls" .-> Runner
Runner -->|per-tool policy| Gate{Permission?}
Gate -->|auto| Invoke[Invoke typed cap]
Gate -->|consent| Prompt[Prompt user y/n]
Prompt --> Invoke
Gate -->|stepUp| Broker[AuthorityBroker step-up]
Broker --> Invoke
Gate -->|forbidden| Refuse[Refuse, feed error back]
Invoke --> Services[Session caps: files, net, spawn, status...]
Invoke --> Audit[AuditLog]
Services -. "result" .-> Runner
Runner -->|role:tool result| ModelSvc
```
Two principals matter in the capOS-side runner model:
- **Agent runner.** Holds the session cap bundle (terminal, home, logs,
launcher, approval, model client, etc.). Runs the user-facing loop,
talks to the model, applies per-tool permission policy, executes tool
calls against its held caps, streams results back to the model, and
writes audit. This is the natural daily driver — either the native
shell in "agent mode" or a sibling process launched from the shell.
- **Model service.** Holds weights, an optional accelerator session, and
an optional narrow outbound `HttpEndpoint` for remote backends. Sees
conversation messages; emits text and tool calls. Has no session, no
tools, no spawn authority.
The kernel does not need a "model" or "agent" concept. Everything here
is ordinary capabilities, processes, and ring traffic.
### Browser Agent UI
In a web shell, the agent itself may be the UI. Browser JavaScript may render
the conversation, call a provider LLM API directly, receive structured tool
calls, and feed tool results back into the model. That mode exists for
latency, provider-native browser SDKs, and richer UI composition.
It still does not give browser JavaScript raw capOS capabilities:
```mermaid
flowchart LR
User[User] --> BrowserAgent[Browser Agent UI
LLM loop]
BrowserAgent -->|ephemeral provider token| Provider[LLM Provider API]
Provider -. "text + tool calls" .-> BrowserAgent
BrowserAgent -->|ToolRequest| Gateway[WebShellGateway
ToolProxy]
Gateway --> Broker[AuthorityBroker]
Gateway --> Audit[AuditLog]
Gateway --> Services[Session caps: files, net, spawn, status...]
Services -. "typed result" .-> Gateway
Gateway -. "ToolResult" .-> BrowserAgent
BrowserAgent -. "tool result" .-> Provider
```
Authority split:
- **Browser agent UI.** Owns presentation, local conversation state, user
gestures, optional browser media APIs, and direct provider session state.
It holds no capOS caps, no session caps, no tool caps, and no provider
long-lived credentials.
- **WebShellGateway tool proxy.** Owns the authenticated web transport and
the server-side reference to the session bundle. It exposes the current
tool descriptor snapshot to the browser, accepts structured `ToolRequest`
values, validates them against the session, enforces broker policy and
consent/step-up, invokes the real capOS capabilities, and writes audit.
- **Provider.** Sees prompts and tool results only when broker policy allows
direct browser provider use for the session's confidentiality profile.
The browser-agent model is therefore browser-orchestrated but
gateway-enforced. It is not a bearer-capability model and not a shortcut
around the broker.
## The Tool-Use Loop
One capOS-side agent turn:
```text
1. User types a message (or kicks off the first turn from a CLI arg).
2. Runner assembles: system prompt + prior messages + user message +
the set of ToolDescriptor values the session currently advertises.
3. Runner calls LanguageModel.stream(req). Token stream is rendered to
the terminal as it arrives.
4. Model response finishes. It contains text (shown) plus zero or more
ToolCall records (not shown as text; shown as typed tool-call UI).
5. For each ToolCall:
a. Look up the tool by name. If not in the advertised set, reject
with a typed error fed back as a role: tool result.
b. Validate arguments against the tool's paramSchema. Reject if
malformed; feed the validation error back.
c. Check the tool's permission mode:
- auto: proceed.
- consent: render the call + arguments + permission UI;
wait for user y/n. Deny feeds a refusal back.
- stepUp: request a leased narrow cap from the broker,
possibly driving WebAuthn/OIDC step-up. On
success, proceed; on denial, feed back.
- forbidden: reject; feed typed "not permitted in this
session" error back.
d. Invoke the underlying typed capability. Time-box the call.
e. Truncate/redact the result per tool policy, serialize as a
role: tool message keyed to the ToolCall id.
6. If the model emitted tool calls, loop back to step 3 with the
results appended. If it emitted none (or the user interrupted),
this turn ends.
7. Every executed call produces an audit record.
```
One browser-agent UI turn:
```text
1. User interacts with the browser agent UI.
2. Browser agent assembles the prompt, prior messages, and the current
ToolDescriptor snapshot fetched from WebShellGateway.
3. Browser agent calls the provider directly using a broker-minted,
short-lived, provider-scoped token.
4. Provider response streams into the browser. If it emits ToolCall records,
the browser wraps each as a ToolRequest to WebShellGateway.
5. WebShellGateway validates the call against the advertised descriptor,
current session state, nonce/turn binding, quotas, and broker policy.
6. WebShellGateway obtains any required consent or step-up proof, invokes the
underlying capOS capability server-side, writes audit, and returns a
ToolResult.
7. Browser agent feeds the ToolResult back to the provider and continues the
loop until no tool calls remain or the user/gateway cancels.
```
Browser-originated tool requests are untrusted input even when the agent is
the intended UI. The gateway must reject stale descriptors, unknown tools,
argument/schema mismatches, replayed turn ids, requests outside the current
session profile, and any operation whose consent or step-up proof is missing.
Interactive-agent niceties that fall out of this structure:
- **Streaming.** Tokens render live. Tool calls appear as structured
widgets, not as text the user has to parse.
- **Interruption.** Ctrl-C at any point cancels the in-flight inference
(`TokenStream.cancel`) or the in-flight tool call. The runner decides
whether to feed a cancellation message back to the model or end the
turn.
- **Auto vs. consent.** Reading files, listing directories, querying
`SystemStatus`, reading logs — `auto`. Writing files, spawning
processes, changing service state, sending network requests —
`consent`. Destroying data, running a recovery operation, widening
the session's own caps — `stepUp` or `forbidden`.
- **Context management.** When the transcript approaches
`ModelInfo.contextTokens`, the runner can summarise older turns (via
a second `LanguageModel.complete` call) and replace them with a
compact summary message. This is a runner decision, not a kernel or
model feature.
- **Conversation persistence.** A conversation is a list of messages
plus a reference to the runner's session; it can be written to a
`home`-scoped file, resumed later, forked, or compared. Persistence
is an ordinary capability concern, handled by the runner through
whatever `Directory`/`File` cap it holds.
### Agent Mode is a Mode of the Native Shell
The native shell from [shell-proposal.md](shell-proposal.md) becomes the
agent runner. It already holds the session bundle; adding a
`LanguageModel` client cap plus a per-tool permission table gives it
"agent mode". In that mode:
- Plain user input becomes a chat turn.
- `/cap`, `/inspect`, `/exit`, and the other existing direct commands
stay as direct typed invocations that bypass the model.
- Tool descriptors are generated by the same schema reflection the shell
already needs for its capability REPL.
A separate `capos-agent` binary is possible for deployments where agent
mode is the default (think "bare capOS image with no traditional
shell"). It launches from the same login path with the same session
bundle, differs only in the surface presented to the user.
### Web Agent Mode is a Mode of WebShellGateway
For browser-hosted sessions, `WebShellGateway` exposes an agent UI protocol
instead of making the browser a capOS process. The protocol can be JSON over
WebSocket or another web-native framing, but its values mirror
`ToolDescriptor`, `ToolCall`, and `ToolResult` from this proposal.
Gateway responsibilities:
- issue short-lived provider credentials only when policy allows direct
browser LLM access;
- bind the tool descriptor snapshot to a session id, conversation id, turn id,
expiration, and browser connection;
- execute tools only through server-side session caps;
- enforce low-risk consent, mutating consent, and destructive `stepUp`
server-side;
- return redacted/truncated tool results according to tool policy;
- revoke or expire provider tokens where the provider supports it, reject new
tool requests on logout, timeout, tab close, session downgrade, or policy
change, and record any browser-held provider session that can only be
terminated best-effort.
Browser responsibilities:
- render the agent UI and any consent prompts supplied by the gateway;
- preserve provider session state only as long as the gateway session is live;
- submit structured tool requests, never raw capability invocations;
- treat gateway denials, cancellation, and revocation as authoritative.
For mutating or destructive tools, a browser click is not enough by itself.
The gateway needs a fresh server-side consent challenge or a broker-issued
step-up lease tied to the exact tool name, arguments, conversation, turn, and
expiration. Low-risk read-only tools may use auto execution when broker policy
allows.
### Prompt Injection as a First-Class Concern
Model inputs include untrusted data: file contents, log lines, web
pages fetched via a tool call, Aurelian Frontier NPC dialogue, output
from previously executed tool calls. Every such input is wrapped in a
`role: user` or `role: tool` message with explicit provenance, never
concatenated into a `system` prompt. The runner never parses assistant
free text as a command, and the gateway never treats browser-submitted free
text as a capability request. Only structured `toolCalls` / `ToolRequest`
values can reach the tool execution path.
A user can paste `rm -rf /` at the model; the model can repeat it back;
nothing happens, because there is no code path that interprets text as a
command. A web page can instruct the model to exfiltrate secrets; the model
cannot use capOS resources except through the advertised tool set, and
sensitive tools are gated by `consent`/`stepUp`. If the browser agent has
ordinary web-network reachability, broker policy must treat prompts and tool
results as exposed to that browser/provider boundary and deny direct browser
mode for sessions where that is unacceptable.
## Capability Contract
Additions to `schema/capos.capnp` (exact method IDs and argument
packing belong to the implementation PR; the shapes below are the
contract):
```capnp
# Conversation inputs/outputs are plain data. They carry no authority.
struct ChatMessage {
role @0 :Role;
content @1 :Text;
# For role:tool, the id of the ToolCall this message answers.
toolCallId @2 :Text;
# For role:assistant messages that included tool calls, the list
# of calls the model proposed.
toolCalls @3 :List(ToolCall);
}
enum Role {
system @0;
user @1;
assistant @2;
tool @3;
}
struct ToolDescriptor {
name @0 :Text;
description @1 :Text;
# Capability and method the runner will invoke if this tool fires.
interfaceId @2 :UInt64;
methodName @3 :Text;
# JSON-Schema or equivalent describing the argument object.
paramSchema @4 :Text;
# Permission mode. Enforced by the runner, surfaced to the model as
# hint metadata so the model can explain or avoid risky calls.
permission @5 :PermissionMode;
# Tool category for audit and policy filters.
category @6 :Text;
}
enum PermissionMode {
auto @0; # Runner executes without user prompt.
consent @1; # Runner prompts user before execution.
stepUp @2; # Runner requests broker step-up before execution.
forbidden @3; # Advertised for explanation only; never executed.
}
struct ToolCall {
id @0 :Text; # Unique within the conversation.
name @1 :Text;
# Arguments serialised as JSON (or capnp AnyPointer in a later
# revision). The runner validates against paramSchema.
arguments @2 :Text;
}
struct ToolResult {
callId @0 :Text;
outcome @1 :Outcome;
content @2 :Text; # Possibly truncated / redacted by the runner.
error @3 :Text; # Set when outcome != ok.
}
enum Outcome {
ok @0;
refusedByPolicy @1;
deniedByUser @2;
stepUpFailed @3;
executionError @4;
timedOut @5;
cancelled @6;
invalidArguments @7;
unknownTool @8;
}
struct InferenceRequest {
messages @0 :List(ChatMessage);
tools @1 :List(ToolDescriptor);
maxTokens @2 :UInt32;
temperature @3 :Float32;
stopSequences @4 :List(Text);
# Optional JSON-Schema for final-assistant structured output.
responseSchema @5 :Text;
# Stable correlation id for audit.
nonce @6 :Data;
}
struct InferenceResponse {
message @0 :ChatMessage; # role:assistant, may include toolCalls.
usage @1 :TokenUsage;
finishReason @2 :FinishReason;
}
interface LanguageModel {
info @0 () -> (info :ModelInfo);
complete @1 (req :InferenceRequest) -> (resp :InferenceResponse);
# Streaming variant emits token chunks and tool-call deltas as they
# are decoded. Cancellation aborts decoding.
stream @2 (req :InferenceRequest) -> (stream :TokenStream);
}
interface TokenStream {
next @0 () -> (chunk :StreamChunk, done :Bool);
cancel @1 () -> ();
}
struct StreamChunk {
textDelta @0 :Text;
toolCallDelta @1 :ToolCallDelta; # partial structured tool call
}
interface TextEmbedder {
info @0 () -> (info :ModelInfo);
embed @1 (texts :List(Text)) -> (vectors :List(Vector));
}
struct ModelInfo {
id @0 :Text; # Content-addressed weight digest + arch tag.
displayName @1 :Text;
arch @2 :Text; # "llama", "qwen", "phi", etc.
contextTokens @3 :UInt32;
outputTokens @4 :UInt32;
backend @5 :Text; # "local-cpu", "local-gpu", "remote-openai", ...
quantisation @6 :Text; # "fp16", "q4_k_m", ...
supportsTools @7 :Bool;
}
# Administrative surface. Not granted to normal sessions.
interface ModelCatalog {
list @0 () -> (models :List(ModelInfo));
openLanguageModel @1 (id :Text) -> (model :LanguageModel);
openEmbedder @2 (id :Text) -> (embedder :TextEmbedder);
}
interface ModelAdmin {
loadWeights @0 (source :ReadOnlyFile, info :ModelInfo) -> (id :Text);
unload @1 (id :Text) -> ();
setBackendPolicy @2 (policy :BackendPolicy) -> ();
}
```
The web-shell protocol should expose a non-capability tool proxy with the same
data shapes. Exact framing belongs to the `WebShellGateway` milestone:
```text
describeTools(session, conversation) -> List(ToolDescriptor)
invokeTool(session, conversation, turn, descriptorSnapshot, ToolCall)
-> ToolResult
cancelTurn(session, conversation, turn) -> ()
```
This is intentionally not a `LanguageModel` method and not a capOS capability
handle passed to the browser. It is an authenticated web transport endpoint
whose implementation invokes real session caps only after gateway/broker
checks pass.
### What is deliberately absent
- No method on `LanguageModel` accepts a capability argument. The model
never holds a live cap to a user resource.
- No method returns a capability that could be invoked outside the
model service (`TokenStream` is the one exception and is scoped to
the current response).
- No "run this tool for me" method on `LanguageModel` or any model
service. Tool execution is the runner's or gateway tool proxy's job. The
model only *names* tools.
- No `PlannerAgent` / `ActionPlan` / dispatcher interface. Planning, if
it happens, is something a model does inside one of its responses; it
is not a separate typed product.
- No "agent shell interface" served by the model. In the capOS-side model,
the shell is the runner and capability holder; in the browser-agent model,
`WebShellGateway` is the capability holder.
## The Agent Runner
This section describes the capOS-side runner. Browser-hosted sessions use the
`WebShellGateway` tool proxy described above instead of placing the runner and
session caps in browser JavaScript.
The runner is an ordinary userspace process (native shell in agent
mode, or `capos-agent`) that holds:
- The session cap bundle, unchanged from the shell proposal.
- A `LanguageModel` client cap issued by the broker.
- A `ModelInfo` read-only view for rendering model identity.
- A `ConversationStore` cap (when one exists) for persistence.
It does *not* hold `ModelCatalog` or `ModelAdmin` — those are
administrative. If a session wants to switch models mid-run, the
broker issues a new `LanguageModel` cap.
### Building the Tool Table
On startup (and after any cap-set change), the runner walks its own
session bundle and produces `ToolDescriptor` values through schema
reflection over the advertised capabilities' interfaces. It applies
the broker-supplied **per-tool permission map** keyed by
`(category, methodName)`:
```text
read-only -> auto
mutating local -> consent
destructive -> stepUp
outbound net -> consent (unless profile allows auto)
admin-class -> forbidden (for non-operator sessions)
```
The runner is free to suppress tools entirely for a given conversation
(for example, never advertise `ServiceSupervisor.restart` for a guest
session, even though the descriptor set could carry a `forbidden`
entry). Suppression is sometimes clearer than presenting an unusable
tool to the model.
### The Loop State Machine
```text
Idle
│ user turn arrives
▼
AssemblingRequest ── tool-descriptor snapshot ─► Inferring (LanguageModel.stream)
▲ │
│ tool result appended │ model finishes
│ ▼
ExecutingCalls ◄─── one call at a time ───────── HasToolCalls?
│ per call: gate → execute → audit │ no
└──────────────────────────────────────────┐ ▼
│ Idle
▼
(any denial / cancel is
an outcome fed back)
```
Timeouts are enforced at three levels: per-tool (so a slow capability
does not block the loop forever), per-turn (bounded number of
iterations to prevent runaway), and per-session (token and
wall-clock budgets from the broker).
### Conversation State
A conversation is `List(ChatMessage)` plus a `ModelInfo.id`, the
effective `ToolDescriptor` table at each turn, and the audit trail.
The runner keeps it in its own process memory during a session and may
persist it through a `ConversationStore` cap (when that exists; see
open questions). No conversation state lives in the model service; the
service is stateless across requests.
## The Built-in Local Model
capOS ships with a small local language model so that:
- First boot has a working agent without remote network.
- The adventure and chat demos can have a real local NPC brain rather
than hard-coded strings.
- Offline and air-gapped deployments remain viable.
- The capability surface has a real local implementation to validate
against before remote backends are wired up.
### Constraints
- **Size budget.** A 1–3 B parameter quantised model (`q4_k_m`-class)
fits in 0.7–2.0 GiB. That is too large for `manifest.bin` embedding
(2.75 MiB cap) and forces the ISO filesystem path — see the Boot
Binary ISO Layout item in `docs/backlog/hardware-boot-storage.md`. Weights
are the first non-binary consumer of the ISO file path.
- **Tool calling.** The model must be a tool-use-capable instruction
tune (a chat-tuned model without reliable tool-call formatting
cannot drive the loop). `ModelInfo.supportsTools` flags this.
- **Backend.** First implementation is CPU-only, portable Rust
inference. Candidates include `candle` (needs `no_std` survey), a
minimal hand-rolled GGUF loader + matmul kernel, or a vendored
subset of a permissively licensed engine. Final choice is an
implementation decision, not a proposal decision; the capability
surface is implementation-agnostic.
- **Precision.** `q4_k_m` or `q5_k_m` quantised GGUF. fp16 is a later
optimisation gated on either SIMD-friendly CPU support or GPU
acceleration.
- **Context window.** 4 K–8 K tokens at first. Enough for short agent
sessions; long-document summarisation is a later workload that may
require a different model or aggressive runner-side compaction.
- **Attestation.** Weights are signed (see
[cryptography-and-key-management-proposal.md](cryptography-and-key-management-proposal.md))
and the signature is verified at load. The content-addressed digest
becomes the `ModelInfo.id`.
### Boot Flow
1. ISO driver (pending the Boot Binary ISO Layout item in
`docs/backlog/hardware-boot-storage.md`) exposes
`/boot/models/.gguf` as an ordinary file.
2. Kernel or a privileged loader service constructs a read-only
file-backed `MemoryObject` over the weights file. Read-only shared
frames let multiple model worker processes map the same weights
without copies.
3. `model-loader` service (started from the manifest) verifies the
signature, registers the model in `ModelCatalog`, and keeps a
retained handle to the weights `MemoryObject`.
4. On demand, `ModelCatalog.openLanguageModel(id)` spawns (or returns
a handle to) a worker process holding the weights, an inference
kernel, and — if policy allows — a `GpuSession` or a remote
`HttpEndpoint`.
Weights never live in the manifest blob. The ISO layout work is the
prerequisite, and this proposal is its first forcing use case larger
than a few megabytes.
### Page Cache Coupling
Multiple sessions sharing one model benefit from a page cache over the
weights file: the first access faults in, subsequent accesses hit
cache, and the pages are shared read-only across all worker processes.
This is the same primitive that makes ELF text-segment sharing useful,
and it should be implemented once in the ISO/file-backed-memory path
rather than specialised per consumer.
## CapOS-Side Backends
CapOS-side backends sit behind `LanguageModel` / `TextEmbedder`. The worker
process loads exactly one backend per instance. Browser direct-provider mode
is a separate web transport mode described below; it is not a
`LanguageModel` worker backend.
### Local CPU
- File-backed read-only weights mapped from ISO or storage.
- No accelerator caps. No network caps.
- Bounded per-call token budget enforced by the worker; broker sets
per-cap quotas.
### Local GPU
- Holds a `GpuSession` from the
[GPU capability proposal](gpu-capability-proposal.md).
- Holds a read-only `MemoryObject` for the weights; uploads to GPU
memory at load time through `GpuBuffer`.
- Still no network. Still no session cap.
### Remote Provider
- Holds one narrow `HttpEndpoint` scoped to a single provider origin
(for example an Ollama instance on the local network, or an external
API gateway). The endpoint is issued by the broker; the model worker
cannot widen it.
- Holds provider credentials only as token-typed capabilities (OAuth
`AccessToken` wrapped as a cap, never exposed as a bearer string —
see [OIDC and OAuth2 proposal](oidc-and-oauth2-proposal.md)).
- The model worker process is still the principal that talks to the
remote; the runner never sees provider credentials.
- Treated as untrusted: outbound request/response logging is mandatory
when operator policy requires audit of off-device inference.
### NPU / Future Accelerators
Same shape. Add a scoped `NpuSession` cap analogous to `GpuSession`
when the hardware abstraction for it exists.
## Browser Direct-Provider Mode
- Browser receives only a broker-minted ephemeral credential scoped to one
provider, model/config, session, conversation, and short expiration.
- The credential contains no capOS capability material and cannot be exchanged
for session caps.
- The browser may run the provider's JavaScript/WebRTC/WebSocket client and
orchestrate the LLM loop.
- Tool execution still goes through `WebShellGateway`'s tool proxy; provider
tool declarations must match the gateway-advertised descriptors for that
turn.
- Broker policy may deny this mode for sessions whose prompts, tool results,
labels, or audit requirements cannot leave the capOS-side trust boundary.
- Logout, tab close, timeout, or session downgrade authoritatively closes the
capOS session and rejects future tool requests. Provider token/session
revocation is authoritative only when the provider exposes a server-side
revocation or session-close API; otherwise it is best-effort and must be
audited as such.
## Policy and the Broker
`AuthorityBroker` gates every model interaction:
- Which session profiles get a `LanguageModel` cap at all (operator:
yes; anonymous: usually no; guest: local-only, no remote providers).
- Which backend resolves an `openLanguageModel(id)` call for this
session (local-only for unclassified work; remote permitted for
operators who opted in and passed step-up auth).
- Rate and token-budget limits per session and per principal.
- The **per-tool permission map** the runner applies when building
its tool table, or that `WebShellGateway` applies before publishing
descriptor snapshots to a browser agent. This is the main policy knob:
an anonymous session might get only read-only tools as `auto`; an
operator session gets `consent` on mutating tools and `stepUp` on
destructive ones.
- Outbound-network egress policy for remote backends.
- Whether direct browser provider access is allowed for this session, and
which prompts, transcripts, tool descriptors, and tool results may cross
that browser/provider boundary.
- PII / confidentiality labels: a session labelled MAC/MIC-high may be
denied remote inference entirely because prompts would cross the
confidentiality boundary (see
[formal-mac-mic-proposal.md](formal-mac-mic-proposal.md)).
The broker's decisions are recorded in audit. The model service itself
performs no policy checks — it is an execution backend.
## Audit and Provenance
Every executed tool call audit record includes:
- Session ID, principal, conversation ID, turn index, tool-call ID.
- Model identity (`ModelInfo.id`), backend, request nonce.
- Runner location (`capos-side` or `browser-agent-ui`) and gateway session id
when a browser agent proposed the call.
- Advertised tool descriptor at the time of the call (name,
`paramSchema`, permission mode).
- Exact typed arguments.
- Permission decision (auto, consented, denied, step-up-succeeded,
step-up-failed, forbidden).
- Tool outcome, truncated result hash, and error if any.
Optional per-session conversation-level records capture message
metadata (role, timestamp, length, hash) without requiring full prompt
content to be stored — the classification policy decides how much
content is retained.
This lets an operator answer "what did the agent do on my behalf last
week, which model produced each call, and which tools were visible"
without replaying prompts from logs the model service does not hold.
## Threat Model
Assumed hostile:
- Prompts, retrieved documents, web pages, and tool-call outputs.
- Model weights from unknown sources (mitigated by weight signing and
`ModelInfo.id` attestation).
- The model worker process itself — treated as a semi-trusted data
transformer, isolated by its narrow CapSet.
- Browser JavaScript, browser extensions, DOM state, browser-held provider
sessions, and browser-agent UI code. They may be the intended user
interface, but their tool requests are untrusted inputs to the gateway.
Assumed trustworthy (with attestation):
- The kernel, the capOS-side runner when used, `WebShellGateway`'s server-side
tool proxy, the broker, the ISO driver, and the loader.
Out of scope (covered by other proposals or tracks):
- Side-channel leakage through cache timing on shared accelerators —
follow work on GPU tenant isolation in the GPU proposal.
- Model-backdoor detection — an ecosystem problem, not a kernel one;
capOS only guarantees that a compromised weights file cannot escape
its worker process's CapSet.
## Integration with Existing Workloads
- **Adventure demo.** NPC processes can hold a narrow `LanguageModel`
cap scoped to small prompt budgets, producing in-character lines
instead of canned strings. Chat rooms can feed the demo through a
runner variant without session-level tools.
- **Boot-to-shell first-use.** First boot can offer an agent-assisted
setup flow ("help me configure the network stack") once the runner
is wired up and the operator session profile includes the model cap
and the right tool permission map.
- **Log and metric summarisation.** `LogReader` becomes a
`consent`-gated tool in the runner's tool table. The model asks for
"last hour of auth errors"; the runner executes, truncates, feeds
back. The model never holds `LogReader` itself.
- **Semantic search over directories.** `TextEmbedder` + a vector
index service (future) lets `home`/`docs`-scoped search work through
a `search` tool advertised by the runner, without ambient file
access for the model.
## Implementation Phases
### Phase 0 — Prerequisites
- ISO 9660 driver + file-backed read-only `MemoryObject`
(`docs/backlog/hardware-boot-storage.md` and the follow-on file-backed
memory work).
- Page cache over file-backed memory.
- `HttpEndpoint` scoped-origin fetch (networking proposal Phase B).
- `AuthorityBroker` and `ApprovalClient` wiring (shell proposal
implementation plan step 4).
- Schema reflection sufficient to build `ToolDescriptor` values.
### Phase 1 — Capability scaffolding
- Add `LanguageModel`, `TextEmbedder`, `ModelInfo`, `ModelCatalog`,
`ModelAdmin`, `ToolDescriptor`, `PermissionMode`, `ToolCall`,
`ToolResult`, `InferenceRequest`, `InferenceResponse`,
`StreamChunk`, `TokenStream` to `schema/capos.capnp`.
- Generate bindings via existing `tools/capnp-build`.
- Stub `language-model` service process with a deterministic
canned-tool-call backend so the runner loop can be exercised
without any real inference.
- `make run-agent` smoke: shell in agent mode runs a scripted
conversation through the stub, exercises auto / consent / stepUp
/ forbidden gates, and exits cleanly.
### Phase 2 — Built-in local model
- Choose a CPU inference engine and vendor it.
- Ship one tool-use-capable quantised model in
`iso_root/boot/models/` as a content-addressed GGUF with a
signature.
- Loader service verifies signature, maps weights, registers in
`ModelCatalog`.
- First real tool-use loop with a local model.
### Phase 3 — Runner features
- Streaming render into `TerminalSession` with interrupt support.
- Context-budget compaction (summarise older turns via a secondary
inference call).
- Per-tool consent UI.
- Audit integration.
- Conversation persistence through a `ConversationStore` cap.
- `WebShellGateway` tool proxy: descriptor snapshots, turn binding,
replay rejection, server-side consent/step-up enforcement, and
browser-agent-proposed audit records.
### Phase 4 — Backends
- GPU backend wired through `GpuSession`.
- Remote-provider backend wired through `HttpEndpoint` + token-typed
capability. One concrete provider (for example local Ollama) as
the proof.
- Broker policies for backend selection.
- Browser direct-provider mode: broker-minted ephemeral credentials,
short token expiry, provider revocation/close when supported, audited
best-effort teardown otherwise, and a web-agent smoke that proves
browser-orchestrated tool calls are executed only through
`WebShellGateway`.
### Phase 5 — Hardening and features
- Structured-output (JSON/capnp) validation against `responseSchema`.
- Embedding-backed retrieval service (`TextEmbedder` + vector store).
- Prompt redaction for MAC/MIC-high sessions.
- Audit replay tooling.
- Step-up integration with the broker's WebAuthn/OIDC paths.
### Phase 6 — Applications
- Agent-assisted adventure NPCs with per-NPC caps.
- Agent-assisted first-boot setup flow.
- Log-summarisation and monitoring assistant.
- Optional: agent mode over the POSIX compatibility layer, once that
exists.
## Dependencies
Hard prerequisites:
- ISO filesystem driver and file-backed `MemoryObject`
(`docs/backlog/hardware-boot-storage.md` plus file-backed memory follow-on).
- `AuthorityBroker`, `ApprovalClient` (shell proposal plan step 4).
- `WebShellGateway` authenticated transport and server-side session tracking.
- `ProcessSpawner` with exact-grant child launch (done).
- Schema reflection / `SchemaRegistry`.
- Cap'n Proto schema evolution tooling (done).
Soft / enables richer behaviour:
- GPU capability proposal for GPU backend.
- OIDC/OAuth2 proposal for remote-provider credentials and step-up
authentication.
- WebAuthn/passkey support for browser step-up on destructive tools.
- Cryptography/KMS proposal for weight signing.
- System monitoring proposal for audit integration.
- Formal MAC/MIC proposal for high-confidentiality session policy.
## Non-Goals
- No kernel-side model awareness.
- No ambient "AI" privilege anywhere.
- No model-issued capabilities.
- No long-lived bearer-token exposure to the runner or browser. Browser-agent
UI mode may use only short-lived provider-scoped credentials.
- No promise that any particular model size, license, or benchmark
score ships in-tree — the choice is an implementation decision
gated by the trusted-build-inputs process.
- No plan/approve/execute pipeline as the primary interaction
(explicitly superseded by the tool-use loop).
- No claim that capOS offers strong defences against model-internal
adversarial attacks (jailbreaks, refusal bypass). The capability
model defends the *system*, not the model's own behaviour.
## Open Questions
- Should tool arguments be JSON (matches provider ABIs like OpenAI
tools / Anthropic tools) or capnp `AnyPointer` (matches capOS
wire format)? Proposed: start with JSON for compatibility with
remote providers and because local GGUF tool-use tunes are
JSON-trained, and add a capnp fast path later.
- How are conversations named, persisted, and resumed? A
`ConversationStore` cap with TTL is the sketch, but the storage
proposal needs an update before this is concrete.
- What is the smallest credible local model that still drives the
tool-use loop reliably for capOS-internal tasks (file edits, status
summaries, NPC dialogue)? Below a threshold, better to ship no
default model and require explicit configuration.
- How should streaming back-pressure compose with ring `cap_enter`
completion limits? A single response can produce many small CQEs.
- When consent prompts pile up in a long turn, how should the runner
offer "approve-once" vs. "approve-for-this-turn" vs.
"approve-for-this-session" without widening authority beyond what
the user intended? A per-session "always allow this tool" allow-list,
cleared at session end, is a reasonable starting point.
- Should the runner ever let the model *read* tool descriptors for
tools it cannot execute (`forbidden`), so the model can explain why
it can't help, or should those be suppressed entirely?
- Does the built-in model warrant its own trust anchor in the weights
signing chain, or should it share the system trust store? Likely
share, with a dedicated key purpose (see cryptography proposal).
- Which web-shell profiles should allow browser-agent UI mode by default?
Operator sessions may want it for latency and provider UX; high-label or
audit-strict sessions should probably force capOS-side provider mediation.
- How should the gateway prove fresh user presence for browser-agent
approvals without trusting arbitrary JavaScript events? WebAuthn/passkey
step-up handles destructive tools; low-risk consent still needs a concrete
freshness rule.