Proposal: Error Handling for Capability Invocations

How capOS communicates errors from capability calls back to userspace processes.

This proposal defines a two-level error model: transport errors (the invocation mechanism itself failed) and application errors (the capability processed the request and returned a structured error). The design aligns with Cap’n Proto’s own exception model and the patterns used by seL4, Zircon, and other capability systems.

Note (2025-06): This proposal was written when cap_call was the synchronous invocation syscall. Since then, cap_call has been replaced by the shared-memory capability ring + cap_enter syscall. The two-level error model and CapException schema remain valid, but the delivery mechanism changes: transport errors and application errors are communicated through CQE result fields (status code + result buffer), not syscall return values. The “Syscall Return Convention” section below describes the original cap_call convention; a future revision should map these error codes to CQE status fields instead.

Current CQE Error Namespace

The capability ring uses signed 32-bit CapCqe.result values. Non-negative values are opcode-specific success results; negative values are kernel transport errors defined in capos-config/src/ring.rs:

This is deliberately a small transport namespace. Interface-specific failures should be encoded in the result payload once the target capability successfully handles the request.

Transfer-related transport mapping (3.6.0 ABI slice)

• CAP_ERR_TRANSFER_NOT_SUPPORTED is used for transfer-bearing SQEs that the kernel currently dispatches but does not yet process (xfer_cap_count != 0 on kernels where sideband transfer is off).
• CAP_ERR_INVALID_TRANSFER_DESCRIPTOR is used for structurally validly dispatched transfer SQEs where transfer metadata is malformed:
  • descriptor transfer_mode is not exactly CAP_TRANSFER_MODE_COPY or CAP_TRANSFER_MODE_MOVE;
  • any descriptor reserved bits are set;
  • any descriptor _reserved0 field is non-zero;
  • descriptor region placement (addr + len) is misaligned;
  • descriptor range overflows or cannot be safely bounded.
• CAP_ERR_TRANSFER_ABORTED is reserved for transaction failure after partial transfer side effects are prepared and must not be observed (all-or-nothing rollback boundary).
• CAP_ERR_INVALID_REQUEST remains for non-transfer transport malformation (unsupported opcodes for today, unsupported SQE fields not part of the transfer path, and malformed result/payload buffer pairs).

Problem Statement

Currently, cap_call returns u64::MAX on any error and prints the details to the kernel serial console. The userspace process receives no information about what went wrong – it cannot distinguish “invalid capability ID” from “method not implemented” from “out of memory inside the service.”

Every other capability system separates transport-level errors (bad handle, message validation failure) from application-level errors (the service processed the request and returned a meaningful error). capOS needs both.

Background: How Other Systems Do This

Cap’n Proto RPC Protocol

The Cap’n Proto RPC specification defines an Exception type in rpc.capnp:

struct Exception {
  reason @0 :Text;
  type @3 :Type;
  enum Type {
    failed @0;        # deterministic failure, retrying won't help
    overloaded @1;    # temporary resource exhaustion, retry with backoff
    disconnected @2;  # connection to a required capability was lost
    unimplemented @3; # method not supported by this server
  }
  trace @4 :Text;
}

These four types describe client response strategy, not error semantics. The capnp Rust crate maps them to capnp::ErrorKind::{Failed, Overloaded, Disconnected, Unimplemented}.

Cap’n Proto’s official philosophy (from KJ library and Kenton Varda’s writings): exceptions are for infrastructure failures, not application semantics. Application-level errors should be modeled as unions in method return types.

Capability OS Error Models

Common pattern: a small kernel error code set for transport failures, combined with service-specific typed errors for application failures.

POSIX errno: Why Not

POSIX errno is a global flat namespace of ~100 integers that conflates transport errors (EBADF) with application errors (ENOENT). In a capability system:

• EACCES/EPERM don’t apply – if you have the capability, you have permission; if you don’t, you can’t even name the resource.
• A global error namespace conflicts with typed interfaces where errors should be scoped to the interface.
• No room for structured information (which argument was invalid, how much memory was needed).
• Not composable across trust boundaries – a callee’s errno has no meaning in the caller’s address space without explicit serialization.

Design

Principle: Two Levels, One Wire Format

Level 1 – Transport errors are returned in the syscall return value. These indicate that the capability invocation mechanism itself failed before the target CapObject was reached. No result buffer is written.

Level 2 – Application errors are returned as capnp-serialized messages in the result buffer. The capability was found and dispatched; the implementation returned a structured error. The syscall return value distinguishes this from a successful result.

Both levels use Cap’n Proto serialization for the error payload (level 2 always, level 1 when there’s a result buffer available). This keeps one parsing path in userspace.

Syscall Return Convention

The cap_call syscall (number=2) currently returns:

• 0..N – success, N bytes written to result buffer
• u64::MAX – error (undifferentiated)

New convention:

The transport error codes are a small closed set (like seL4’s 11 values). New transport errors can be added, but the set should remain small and stable.

CapException Schema

Add to schema/capos.capnp:

enum ExceptionType {
    failed @0;
    overloaded @1;
    disconnected @2;
    unimplemented @3;
}

struct CapException {
    type @0 :ExceptionType;
    message @1 :Text;
}

This mirrors Cap’n Proto RPC’s Exception struct. The four types match capnp::ErrorKind and describe client response strategy:

• failed – deterministic failure, retrying won’t help. Covers invalid arguments, invariant violations, deserialization errors, and any capnp::ErrorKind variant not in the other three categories.
• overloaded – temporary resource exhaustion (out of frames, table full). Client may retry with backoff.
• disconnected – the capability’s backing resource is gone (device removed, process exited). Client should re-acquire the capability.
• unimplemented – unknown method ID for this interface. Client should not retry.

The message field is a human-readable string for diagnostics/logging. It must not contain security-sensitive information (internal pointers, kernel addresses) since it crosses the kernel-user boundary.

Application-Level Errors in Interface Schemas

Following Cap’n Proto’s philosophy, expected error conditions that a caller should handle programmatically belong in the method return type, not in the exception mechanism.

Example – FrameAllocator can legitimately run out of memory:

struct AllocResult {
    union {
        ok @0 :UInt16;       # result-cap handle index for a MemoryObject
        outOfMemory @1 :Void;
    }
}

interface FrameAllocator {
    allocFrame @0 () -> (result :AllocResult);
    allocContiguous @1 (count :UInt32) -> (result :AllocResult);
}

The caller can pattern-match on the result union without parsing an exception. This is the Zircon/FIDL model: transport errors at the syscall layer, application errors as typed return values.

When to use each:

Kernel Implementation

CapObject trait change

The ring SQE does not carry a caller-supplied interface ID. The trait shape below keeps interface selection out of capability implementations because each capability entry owns one public interface:

#![allow(unused)]
fn main() {
pub trait CapObject: Send + Sync {
    fn interface_id(&self) -> u64;
    fn label(&self) -> &str;
    fn call(
        &self,
        method_id: u16,
        params: &[u8],
        result: &mut [u8],
        reply_scratch: &mut dyn ReplyScratch,
    ) -> capnp::Result<CapInvokeResult>;
}
}

Implementations serialize directly into the caller’s result buffer and return a completion containing the number of bytes written, or Pending for async endpoint calls. Dispatch uses the interface assigned to the target capability entry; normal CALL SQEs do not need to repeat that interface ID. capnp::Error carries ErrorKind with the four RPC exception types. The kernel’s dispatch handler converts Err(capnp::Error) into a serialized CapException message and writes it to the result buffer.

Syscall handler changes

In cap_call(), the error path changes from:

#![allow(unused)]
fn main() {
Err(e) => {
    kprintln!("cap_call: ... error: {}", e);
    u64::MAX
}
}

to:

#![allow(unused)]
fn main() {
Err(CapError::NotFound) => ECAP_NOT_FOUND,
Err(CapError::StaleGeneration) => ECAP_NOT_FOUND,
Err(CapError::InvokeError(e)) => {
    // Serialize CapException to result buffer
    let exception_bytes = serialize_cap_exception(&e);
    if result_ptr != 0 && result_capacity >= exception_bytes.len() {
        copy_to_user(result_ptr, &exception_bytes);
        ECAP_APPLICATION_ERROR
    } else {
        ECAP_APPLICATION_ERROR_NO_BUFFER
    }
}
}

The serialize_cap_exception function maps capnp::ErrorKind to ExceptionType:

This matches how capnp-rpc maps exceptions to the wire format.

Userspace API

The init crate (and future userspace libraries) wraps cap_call in a helper that interprets the return value:

#![allow(unused)]
fn main() {
pub enum CapCallResult {
    Ok(Vec<u8>),
    Exception(ExceptionType, String),
    TransportError(TransportError),
}

pub enum TransportError {
    InvalidCapability,
    InvalidBuffer,
    ParamsTooLarge,
}

pub fn cap_call(
    cap_id: u32,
    method_id: u16,
    params: &[u8],
    result_buf: &mut [u8],
) -> CapCallResult {
    let ret = sys_cap_call(cap_id, method_id, params, result_buf);
    match ret {
        ECAP_NOT_FOUND => CapCallResult::TransportError(TransportError::InvalidCapability),
        ECAP_BAD_BUFFER => CapCallResult::TransportError(TransportError::InvalidBuffer),
        ECAP_PARAMS_TOO_LARGE => CapCallResult::TransportError(TransportError::ParamsTooLarge),
        ECAP_APPLICATION_ERROR => {
            let (typ, msg) = deserialize_cap_exception(result_buf);
            CapCallResult::Exception(typ, msg)
        }
        ECAP_APPLICATION_ERROR_NO_BUFFER => {
            CapCallResult::Exception(ExceptionType::Failed, String::new())
        }
        n => CapCallResult::Ok(result_buf[..n as usize].to_vec()),
    }
}
}

Future: Batched Calls

When capOS adds batched capability invocations (async rings, pipelining), each request in the batch gets its own result status. The same two-level model applies per-request:

• Transport error for the batch envelope (invalid ring descriptor, bad capability table) fails the whole batch.
• Per-request transport errors (individual bad cap_id) fail that request.
• Application errors are per-request, written to each request’s result slot.

This matches how NFS compound operations and JSON-RPC batch requests work: a transport error on the batch vs per-operation results.

What This Does NOT Cover

• Error logging/tracing infrastructure. How errors get collected, aggregated, or displayed is a separate concern. The kernel currently prints to serial; a future ErrorLog capability could capture structured error streams.
• Retry policy. The ExceptionType hints at retry strategy (overloaded -> retry, failed -> don’t), but the retry logic itself belongs in userspace libraries, not the kernel.
• Error propagation across capability chains. When capability A calls capability B which calls capability C, and C fails – how does the error propagate back through A? This is a concern for the IPC and capability transfer stages (Stage 6+). The current proposal handles the single-hop case.
• Transactional semantics. Whether a failed operation has side effects (partial writes, allocated-but-not-returned frames) is per-capability semantics, not a kernel-level concern.

Migration Path

Phase 1: Transport error codes (minimal, no schema changes)

Change cap_call to return distinct error codes instead of u64::MAX for all failures. Update the init crate to interpret them. No new schema types needed – application errors still use u64::MAX - 3 but without a structured payload (treated as opaque failure).

This is backward-compatible: existing userspace code that checks == u64::MAX sees different values for different errors, but any >= u64::MAX - 255 check catches all errors.

Phase 2: CapException serialization

Add ExceptionType and CapException to the schema. Implement serialize_cap_exception in the kernel. Update init to deserialize and display errors. Now userspace gets the exception type and message string.

Phase 3: Per-interface application errors

As interfaces mature, add typed error unions to method return types for expected error conditions. FrameAllocator::allocFrame returns AllocResult instead of bare UInt64. The exception mechanism remains for unexpected failures.

Design Rationale

Why mirror capnp RPC’s Exception type instead of inventing our own? Cap’n Proto already defines a well-thought-out exception taxonomy. The four types (failed, overloaded, disconnected, unimplemented) map directly to capnp::ErrorKind in Rust. Using the same vocabulary means capOS capabilities can eventually participate in capnp RPC networks without translation. It also means the Rust compiler enforces exhaustive matching on ErrorKind variants that matter.

Why not put error codes in the syscall return value only (like seL4)? seL4’s 11 error codes work because seL4 kernel objects are simple and fixed-function. capOS capabilities are arbitrary typed interfaces – a file system, a network stack, a GPU driver. The error vocabulary is open-ended. Encoding all possible errors as syscall return values would either require an ever-growing enum (fragile) or lose information (back to errno’s problems). The capnp-serialized CapException in the result buffer gives unbounded expressiveness without changing the syscall ABI.

Why not use capnp exceptions for everything (skip the transport error codes)? Because transport errors happen before the capability is reached. There’s no CapObject to serialize an exception. The kernel would have to synthesize a capnp message on behalf of a non-existent capability, which is wasteful and semantically wrong. A small integer return code is cheaper and more honest about what happened.

Why not define a generic Result(Ok) wrapper in the schema? Cap’n Proto generics only bind to pointer types (Text, Data, structs, lists, interfaces), not to primitives (UInt32, Bool). A Result(UInt64) for allocFrame wouldn’t work. Per-method result structs with unions are more flexible and don’t hit this limitation. The cost is a bit more schema boilerplate, which is acceptable given that capOS has a small number of interfaces.

Why string-based messages (like Plan 9) instead of structured error fields? String messages are adequate for diagnostics and logging. Structured error data belongs in the typed return unions (Phase 3), where the schema enforces what fields exist. Putting structured data in CapException would duplicate the schema’s job and encourage using exceptions for flow control, which Cap’n Proto explicitly warns against.