Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

IPC and Endpoints

Endpoints let one process serve capability calls to another process without adding a separate IPC syscall surface. The same ring transport carries ordinary kernel capability calls and cross-process endpoint calls.

Current Behavior

An Endpoint is a kernel capability object with queues for pending client calls, pending server receives, and in-flight calls awaiting RETURN. A service that owns the raw endpoint can receive and return. Importers receive a ClientEndpoint facet that can CALL but cannot RECV or RETURN.

sequenceDiagram
    participant Client
    participant ClientRing as Client ring
    participant Endpoint
    participant ServerRing as Server ring
    participant Server
    Server->>ServerRing: submit RECV on raw endpoint
    Client->>ClientRing: submit CALL on client facet
    ClientRing->>Endpoint: deliver params and caller result target
    Endpoint->>ServerRing: complete RECV with EndpointMessageHeader and params
    ServerRing-->>Server: cap_enter returns completion
    Server->>ServerRing: submit RETURN with call_id and result
    ServerRing->>Endpoint: take in-flight target
    Endpoint->>ClientRing: post caller CQE with result and receiver metadata
    ClientRing-->>Client: wait returns matching completion

If a CALL arrives before a RECV, the endpoint queues bounded params. If a RECV arrives before a CALL, the endpoint queues the receive request. Delivered calls move into the in-flight queue until the server returns or cleanup cancels them.

Those bounds are currently structural and per endpoint. Endpoint construction preallocates 32 parameter slots at the structural 64 KiB maximum even when a profile selects a smaller logical queue, and the endpoint promised-answer table has a bounded 64-entry system-global proof capacity with a smaller per-caller sublimit. Neither is yet charged as aggregate physical/continuation cost to a process or session ledger. They contain damage but do not provide cross-owner fair admission; see Resource Governance.

Design

Endpoint IPC is capability-oriented. The manifest can export a raw endpoint from one service; importers get a narrowed client facet. This keeps server-only authority out of clients without introducing rights bitmasks.

CALL and RETURN may carry sideband transfer descriptors. Copy transfers insert a new cap into the receiver while preserving the sender. Move transfers reserve the sender slot, insert the destination, then remove the source on commit. RETURN-side transfers append result-cap records after the normal result payload. Cross-session delivery is additionally checked against the cap hold transfer scope: same-session caps fail closed, cross-session-shareable caps may cross, and service-regrant-only caps need a trusted fixed-session regrant path. CALL SQEs may also request field-granular session disclosure. The kernel intersects that request with the invoked cap’s disclosure scope before delivering any subject fields, so a request without scope or scope without a request exposes only the default opaque caller-session metadata.

Legacy receiver metadata is stored on cap-table hold edges and delivered to servers with endpoint invocation metadata, so one endpoint can distinguish transitional callers without one object per caller. Some ABI structs still name this field badge; that name is compatibility state, not the normal shared-service authority model. Session-bound invocation context is the replacement model for normal workload paths: every normal process has one immutable session context, endpoint calls expose privacy-preserving caller-session metadata by default, and shared services derive user-facing state from broker-granted capabilities plus service-scoped session references. See Session Context.

Delegated Client Relabeling Containment

The Gate 0 containment rule is narrow: a process that holds an imported ClientEndpoint may delegate that same client identity, but it may not mint a sibling identity by setting another legacy badge during spawn. Endpoint owners and explicit trusted mint paths remain transitional mechanisms for low-level tests. Normal shared services use broker-granted roots/facets plus session-bound invocation context instead of service-object badges.

Normal capos-shell help and smoke expectations must therefore omit arbitrary badge N launch examples. Omitted shell badge syntax preserves the source identity instead of selecting badge zero. Legacy badge syntax may remain reachable only as a debug or hostile-test input, and QEMU coverage for the Telnet blocker must prove both explicit client @name badge N and low-level legacy badge-zero relabel encodings from a nonzero delegated client facet fail closed.

Local Service-Facet Mint

An endpoint owner may mint a client-only ClientEndpoint facet of its own endpoint at runtime with the CAP_OP_MINT_SERVICE_FACET ring opcode, without spawning a process. It stamps an owner-selected interface id and receiver cookie onto the facet and receives it as a result cap in its own cap-table. This is the same authority the spawn serviceObject grant exercises – an owner handing out client facets of its endpoint – expressed as a bounded, local cap-table operation rather than a side effect of ProcessSpawner.spawn. A service that mints one facet per event (for example the network stack minting one facet per accepted socket) no longer has to create a helper process and quiesce shared endpoint traffic across a blocking spawn round trip.

The mint is owner-only, and the opcode gates on ownership directly. It fails closed for a non-owner source (a client, result, or delegated facet – including a ProcessSpawnerEndpointResult cap – whose is_endpoint_owner() is false). This is stricter than the spawn serviceObject grant, which additionally lets a non-owner pass on a Copy facet it already holds to a spawned child: a local mint into the caller’s own table has no pass-on case, so admitting one would let any Copy-facet holder re-mint an identical copy without owning the endpoint. It fails closed with no facet installed on a stale or missing source cap, a too-small or unwritable result buffer, a relabel the resolver rejects, or a cap-table/heap exhaustion, and rolls back the inserted facet on a post-insert copy failure. The minted facet carries no serve authority – CAP_OP_RECV/CAP_OP_RETURN stay reserved to the owner – and inherits, never widens, the owner hold’s transfer scope. The opcode computes the child hold through capos-lib’s resolve_endpoint_facet_delegation and resolve_service_object_facet – with the owner already established – so the minted facet’s shape cannot drift from the spawn path. This is not a revocation or identity-minting path: it hands out a client facet of an endpoint the caller already owns and does not let a non-owner mint a sibling identity, so it stays inside the Gate 0 containment rule above.

Shell-serviced stdio bridges now bind the active child wait to the first opaque live caller-session reference seen on the bridge endpoint. A later call from a different live caller session is answered with an empty result and the child is terminated; transferred caps are released before either normal transfer rejection or caller-session rejection returns. Normal StdIO.close is treated as a clean child close rather than a security rejection.

One-hop promise pipelining lets a client publish an endpoint CALL and a dependent CALL in one batch. The dependent remains unconsumed while the server handles the antecedent. A same-vat kernel-served antecedent resolves inside the caller’s own drain from bounded per-drain scratch. A cross-vat endpoint antecedent cannot resolve there, because another vat only RETURNs after the caller’s drain: the kernel holds the frozen batch in a generation-tagged answer ledger, and when the endpoint RETURN publishes the antecedent CQE and kernel-owned result-cap records it wakes the submitting thread with a private retry result. The runtime cap_enter wrapper consumes that result and re-enters, so the dependent completes in the caller’s own syscall context through the originating batch’s frozen tail; no cross-thread re-drain and no userspace round trip are involved. Server death fails the antecedent with CAP_ERR_SERVER_DIED and the dependent with CAP_ERR_PIPELINE_ANTECEDENT_FAILED; caller teardown drops the generation-bound continuation. Chains deeper than one hop still need a separate ABI.

Invariants

  • Only raw endpoint holders may RECV or RETURN.
  • Imported endpoint caps are ClientEndpoint facets and must reject RECV and RETURN from userspace.
  • Delegating an imported client facet must preserve its server-visible object identity. Only endpoint owners or explicit trusted mint paths may create sibling client identities, and normal services should not treat that identity as user/session authority.
  • The CAP_OP_MINT_SERVICE_FACET runtime mint is owner-only and must fail closed for a non-owner source (including a ProcessSpawnerEndpointResult cap), a stale or missing source cap, an unwritable/too-small result buffer, or an exhausted cap-table/heap, installing no facet on failure. The minted facet must reject RECV/RETURN and inherit – never widen – the owner hold’s transfer scope. It and the spawn serviceObject grant must reach the same authority decision through the shared capos-lib resolvers.
  • Endpoint queues are bounded by call count, receive count, in-flight count, per-call params, and total queued params.
  • Per-endpoint logical bounds must not be described as aggregate process or session quotas. Fixed backing, endpoint fan-out, and shared continuation slots require aggregate owner accounting before availability is claimed.
  • Each in-flight call has a kernel-assigned non-zero call_id.
  • CALL delivery copies params into kernel-owned queued storage before the caller can resume.
  • Move transfer commit must not leave both source and destination live.
  • Transfer rollback must preserve source authority if destination insertion or result delivery fails.
  • Process exit must cancel queued state involving that pid and wake affected peers when possible.
  • Endpoint promised answers must stay globally and per-caller bounded, retain kernel-owned result-cap records, and never dispatch past their originating batch tail.

Code Map

  • kernel/src/cap/endpoint.rs - endpoint queues, client facet, call IDs, cancellation by pid.
  • kernel/src/cap/ring.rs - endpoint CALL/RECV/RETURN dispatch, result copying, deferred cancellation CQEs.
  • kernel/src/cap/transfer.rs - transfer descriptor loading and transaction preparation.
  • capos-lib/src/cap_table.rs - cap-table transfer primitives and rollback.
  • kernel/src/cap/mod.rs - manifest export resolution and client-facet construction.
  • capos-config/src/ring.rs - EndpointMessageHeader, transfer descriptors, transfer result records, endpoint opcodes.
  • demos/capos-demo-support/src/lib.rs - endpoint, IPC, transfer, and hostile IPC smoke routines.
  • demos/endpoint-roundtrip, demos/ipc-server, demos/ipc-client - QEMU smoke binaries.
  • demos/ipc-zerocopy-producer, demos/ipc-zerocopy-consumer - QEMU smoke for the multi-message shared-buffer zero-copy IPC pattern.
  • demos/service-facet-mint - QEMU smoke for the local endpoint-owner service-facet mint (CAP_OP_MINT_SERVICE_FACET).

Validation

  • make run-smoke validates same-process endpoint RECV/RETURN, cross-process IPC, endpoint exit cleanup, legacy badged calls, transfer success/failure paths, and clean halt.
  • make run-spawn validates init-spawned endpoint-roundtrip, server, and client processes.
  • make test-service-facet-mint validates that an endpoint owner mints a client-only facet of its own endpoint locally with no process spawned, that the minted facet refuses RECV and cannot itself mint, and that a client makes a round trip through the runtime-minted facet. A second stage injects a failure at the mint’s per-cap epoch allocation and validates that the caller sees a negative completion, that cap-table occupancy, the cap-slot ledger, and the endpoint reference count are unchanged, and that a retry mints normally.
  • make run-memoryobject-shared validates a one-shot shared-buffer handoff over an endpoint cap transfer.
  • make run-ipc-zerocopy validates the multi-message zero-copy IPC pattern at the substrate level: the producer transfers one MemoryObject to the consumer and then exchanges four record payloads through the shared mapping while endpoint CALLs carry only sequence numbers and checksums. The demo drives raw SQE/CQE construction through capos-demo-support rather than a typed runtime client and uses an ad-hoc seq+checksum framing because the typed SharedBuffer ABI, ring-shaped producer/consumer metadata, and notification primitives are still pending; production services (File.readBuf, BlockDevice.readBlocks, NIC RX/TX rings) will reuse the same MemoryObject substrate through that future surface, not the demo’s framing.
  • cargo test-lib covers cap-table transfer preflight, provisional insertion, commit, rollback, stale generation, and slot exhaustion cases.
  • cargo test-ring-loom covers ring queue behavior that endpoint IPC depends on for completion delivery, including endpoint promise return/wake ordering, CQ-backpressure retry, and teardown suppression.
  • make run-promise-pipeline covers endpoint RETURN success and endpoint-owner death without a userspace round trip between antecedent and dependent.

Open Work

  • Extend Cap’n Proto promise pipelining to chains deeper than one dependent hop.
  • Add a typed SharedBuffer capability surface (ring-shaped producer/consumer metadata, completion signaling, lifetime/quota rules) on top of the raw MemoryObject substrate exercised by make run-ipc-zerocopy.