Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Pingora Architecture and Philosophy: Research Report for capOS

Research on Cloudflare’s Pingora framework and whether capOS high-level interfaces should borrow its shape.

Bottom Line

capOS should build some high-level userspace interfaces inspired by Pingora’s architecture, but should not make Pingora’s HTTP proxy model, callback set, or runtime structure part of the kernel ABI.

The useful idea is not “copy Pingora.” The useful idea is an opinionated library layer that owns repetitive service mechanics and exposes a typed, phase-oriented customization surface to application code. For capOS, that belongs above the capability ring, in userspace libraries and domain services:

  • capos-rt remains the raw transport owner: bootstrap, CapSet, ring client, typed handles, completion matching, release flushing, exception decoding.
  • A future libcapos-service should own service lifecycle mechanics: endpoint receive/return loops, readiness, dependency waiting, shutdown, background tasks, metrics hooks, and graceful handoff.
  • Domain libraries such as libcapos-http, terminal hosts, network services, storage services, and agent services can expose Pingora-style phase hooks for their specific request lifecycle.
  • Kernel capability interfaces should stay narrow, typed, and stable. Do not add a generic Service capability, callback registry, plugin API, or Pingora-like phase machine to the kernel.

This is a “yes, but only at the userspace framework layer” recommendation.

Sources

Primary external sources:

capOS grounding read for this comparison:

What Pingora Is

Pingora is a Rust framework for building programmable network services, especially HTTP proxies. Cloudflare built it after concluding that NGINX’s process/worker architecture and extension model were limiting performance, connection reuse, safety, and feature velocity at Cloudflare scale.

The original design pressure matters:

  • NGINX’s per-worker connection pools harmed reuse as worker count increased. Pingora’s shared multithreaded architecture improved origin connection reuse and reduced new TCP/TLS handshakes.
  • Cloudflare wanted a statically typed, memory-safe implementation language rather than a C core plus Lua extension layer.
  • Cloudflare chose to implement its own HTTP handling rather than rely on an off-the-shelf library because it needed control over non-standard Internet traffic and product-specific behavior.
  • Pingora is a library and toolset, not a finished proxy binary. Users build their own executable around Pingora’s server, service, and proxy APIs.

That last point is the main architectural lesson for capOS: the framework is valuable because it packages the hard reusable mechanics while leaving product logic in typed extension points.

Architecture

Server, Services, and Applications

Pingora’s top-level Server represents one process. It owns configuration, CLI handling, daemonization, signal handling, service startup, graceful shutdown, and zero-downtime upgrade mechanics. A Server hosts multiple services.

A Service is the long-running unit of work. Listening services own one or more endpoints and an application object. Background services run supporting tasks such as discovery, health checks, metrics, or bootstrap logic. Recent Pingora versions also include service dependency metadata, readiness watches, and topological startup ordering.

The layering is deliberately split:

  • Server owns process-level operation.
  • Service owns listener setup, endpoint accept loops, runtime choice, and shutdown propagation.
  • ServerApp handles an established transport stream.
  • HttpServerApp adds HTTP session negotiation and H1/H2 handling.
  • HttpProxy implements the HTTP proxy workflow.
  • User code implements ProxyHttp to customize the proxy phases.

This means the server has no special concept of “proxy” at the root. Proxying is one application shape hosted by the generic service container.

Per-Service Runtime

Each service gets its own runtime/threadpool. Pingora can use Tokio’s normal multi-threaded work-stealing runtime or a “no steal” runtime built from multiple single-threaded Tokio runtimes. The no-steal option exists because work stealing has overhead, while isolated current-thread runtimes can still use multiple cores.

The important design lesson is not the exact runtime. capOS cannot inherit a Tokio process model directly. The lesson is that runtime policy is a service container concern, not application business logic.

Phase-Oriented Proxy Logic

Pingora’s ProxyHttp trait exposes an ordered lifecycle for a proxied request:

  • initialize per-request context,
  • run early and normal request filters,
  • decide whether to serve from cache or go upstream,
  • select an upstream peer,
  • handle connect success or failure,
  • modify the upstream request,
  • process request body chunks,
  • process upstream response headers, body chunks, and trailers,
  • process downstream response headers, body chunks, and trailers,
  • decide retry/failover behavior,
  • report final logging and summaries.

Most filters are optional. A per-request CTX object is created for each request and is passed mutably through the phases. Shared state across requests is ordinary Rust shared state such as Arc, atomics, or locks.

The ergonomics are strong because the framework gives engineers a lifecycle map. Application code overrides the phase where it has policy, while the framework owns parsing, connection setup, pooling, retries, duplex body forwarding, common error response handling, and resource cleanup.

Connection Pooling and Peer Identity

Pingora pools upstream connections automatically after successful requests, but only reuses a connection for the exact same Peer. Its peer identity includes address, scheme, SNI, client certificate, certificate verification behavior, hostname verification, alternate common name, and proxy settings.

The security lesson is broad: resource reuse must be keyed by all attributes that affect authority, identity, confidentiality, and protocol semantics. A connection pool keyed only by address is wrong for a multi-tenant service.

Failure and Retry

Pingora separates connect failure from post-connect proxy failure. It lets application code mark errors retryable, and it documents the idempotency boundary: retrying after the request was sent is not generally safe for non-idempotent methods.

Its common error type carries an error type, source, retry status, optional cause, and context. That mirrors capOS’s existing split between transport errors and typed application exceptions, but Pingora puts more emphasis on whether a high-level operation can be retried.

Operations Are Part of the Framework

Pingora treats startup, daemonization, graceful termination, graceful upgrade, configuration, error logging, Prometheus metrics, readiness, and service dependencies as framework-level concerns.

The zero-downtime upgrade path transfers listening sockets from an old process to a new one and lets existing requests drain during a grace period. That is a specific Linux mechanism, but the higher-level idea maps to capOS live upgrade: stable acceptor or endpoint authority should be retargetable without dropping new work, and old in-flight calls should be allowed to drain when policy says they can.

Philosophy

Pingora’s philosophy is pragmatic, not minimalist:

  • Build a framework, not a monolithic product.
  • Own the hot-path mechanics so users do not reimplement them incorrectly.
  • Expose typed hooks at lifecycle points where policy naturally belongs.
  • Keep common operational behavior in the container rather than every service.
  • Prefer static typing and memory safety for extensibility.
  • Share reusable resources across workers when the safety boundary allows it.
  • Give application code enough control to handle product-specific edge cases.

There is tension in that philosophy. Pingora’s permissive, Internet-facing HTTP goals require supporting odd traffic and complex reuse rules. That flexibility can create security hazards if defaults are too generous or protocol state is not exhausted before reuse.

The 2025 and 2026 Pingora request-smuggling advisories are directly relevant to capOS design. The lesson is not that Pingora is unsafe. The lesson is that high-level frameworks become security-critical because they decide defaults, message framing, cache keys, retry rules, and reuse conditions for their users. capOS libraries should treat those defaults as part of the trusted interface.

Mapping to capOS

What capOS Should Adopt

1. A userspace service framework layer.

capOS already has a low-level transport owner in capos-rt. The next layer should be an opinionated service framework that runs on top of typed capability clients and endpoint server helpers. It should not replace capos-rt; it should use it.

Candidate shape:

  • service lifecycle: init, ready, run, shutdown, drain;
  • dependency waiting: typed readiness handles, not global service names;
  • endpoint serving: generated or handwritten RECV/RETURN loops;
  • background tasks: timers, discovery, health checks, metrics export;
  • graceful handoff: transfer or retarget listener/endpoint authority;
  • structured observability: request summaries, metrics, error suppression policy, and panic boundaries;
  • resource accounting: explicit budgets or donated resources for sessions.

2. Phase-oriented domain libraries.

Pingora-style phases fit domains with real lifecycles:

  • HTTP proxy and fetch service: request filter, route, connect, upstream request, response, body chunks, logging, failover.
  • Terminal host: accept transport, negotiate transport options, authenticate session, spawn shell, proxy terminal I/O, log, cleanup.
  • Storage service: authorize operation, resolve object, choose cache path, perform read/write, commit, audit.
  • Agent service: authenticate caller, bind tool authority, plan invocation, stream outputs, log decision context.

The phase names should be domain-specific. A generic OS-wide phase machine would become vague and hard to secure.

3. Per-request context objects.

Pingora’s CTX model is a good fit for capOS service libraries. Each request or session should have an owned context object dropped at the end of the lifecycle. That context should carry derived policy decisions, peer identity, timing, resource reservations, and transfer state.

This is cleaner than hidden globals and safer than asking later phases to reparse the original request.

4. Resource reuse keyed by authority identity.

Future capOS HTTP/TLS/TCP services should reuse expensive resources, but the pool key must include all security-relevant identity:

  • target address and protocol;
  • TLS SNI, ALPN, certificate policy, and client certificate;
  • authority cap identity or object epoch;
  • caller/session identity if it affects policy;
  • cache namespace or tenant;
  • request transformation policy when it changes what upstream sees.

This is the capOS analogue of Pingora’s strict Peer equality.

5. Operational lifecycle as an API.

The service framework should make readiness, graceful shutdown, and upgrade handoff explicit. That connects to capOS’s future live-upgrade proposal and avoids baking operational behavior into ad hoc service code.

6. Retry semantics as typed policy.

High-level clients should surface retry decisions only where the domain can state idempotency and replay safety. For example, HttpEndpoint.get() can have different retry policy than HttpEndpoint.post(), and a storage write should not be retried unless the interface defines idempotent operation IDs.

What capOS Should Reject

1. Do not make Pingora’s phases kernel concepts.

The kernel should continue to dispatch narrow CapObject methods over the ring. It should not know about request filters, upstream peers, retries, logging phases, or protocol-specific context. Those belong in userspace.

2. Do not add a generic service/plugin capability.

A generic Service.call(phase_id, bytes) or callback registry would weaken capOS’s central design bet: the typed interface is the permission. Use a domain-specific Cap’n Proto interface for authority and a domain-specific library for ergonomics.

3. Do not inherit Pingora’s process model.

Pingora is one unprivileged Linux process hosting multiple services with per-service runtimes. capOS’s isolation model is many processes with explicit capability grants. Service libraries may internally multiplex tasks, but authority boundaries should remain process and capability boundaries.

4. Do not use globals as authority.

Pingora’s ordinary Rust shared-state model is reasonable inside one trusted process. In capOS, cross-service authority must flow through capabilities, not statics, process-wide registries, or global service discovery.

5. Do not ship permissive defaults where explicit policy is needed.

Pingora 0.8.0 removed an insecure cache-key default and hardened HTTP framing. capOS should take this as a rule: cache keys, tenant identity, message framing, body drain behavior, reuse policy, and transfer semantics must be explicit or fail closed.

Concrete capOS Direction

The right decomposition is:

schema/capos.capnp
    Stable authority-bearing interfaces.
    Keep small and domain-specific.

capos-rt
    Raw runtime and transport:
    CapSet, ring, typed handles, release, result caps, exceptions.

libcapos-service
    Generic userspace service container:
    lifecycle, endpoint loops, readiness, shutdown, background tasks,
    metrics, request context, resource budgeting.

domain libraries
    Pingora-like phase APIs where they make sense:
    HTTP/fetch, terminal host, storage, supervisor, agent tools.

init/supervisors
    Compose services by passing capabilities, not by global names.

The first useful application is not the current runtime/Go milestone. The nearest capOS milestone where this should shape implementation is networking Phase B and the Telnet Shell Demo:

  • Keep NetworkManager, TcpListener, TcpSocket, and TerminalSession as narrow capability interfaces.
  • Build the Telnet gateway as a userspace service that uses a lifecycle helper: accept connection, negotiate Telnet, create a socket-backed TerminalSession, spawn shell with exact grants, proxy until exit, log, cleanup.
  • Later, build Fetch and HttpEndpoint services with a Pingora-inspired HTTP lifecycle library rather than exposing raw socket authority to apps.

The first concrete proposal should therefore target terminal/networking lifecycle, not HTTP. This is now tracked in libcapos-service. A useful slice is:

  1. TerminalSessionFromByteStream / byte-stream terminal host;
  2. lifecycle wrapper around accept, session minting, proxying, and cleanup;
  3. metrics plus request/session context hooks;
  4. network service container;
  5. HTTP/fetch services only after the terminal/networking lifecycle proves the authority and cleanup model.

For generated clients, the Pingora lesson argues for generated or handwritten thin wrappers, not raw Cap’n Proto calls everywhere. The wrapper owns:

  • parameter encoding and result decoding;
  • typed application exceptions;
  • retry classification if the interface defines it;
  • result-cap adoption;
  • request summary and metrics hooks.

Risks and Review Rules

Any Pingora-inspired capOS framework should be reviewed against these rules:

  • Extension hooks must receive the narrowest capabilities needed for that phase. Do not hand a broad service object to every hook by convenience.
  • Request context must be lifecycle-owned and dropped deterministically.
  • Pool keys must include all authority and identity fields that affect reuse.
  • Retry policy must be explicit about whether upstream side effects may have happened.
  • Cache-key construction must have no insecure default for multi-tenant data.
  • Protocol parsers must drain or close before reusing a stream.
  • Background tasks must be budgeted and cancellable during service shutdown.
  • Readiness must mean the exported capability is actually ready to serve, not merely that the process started.
  • Generated high-level wrappers must preserve the transport/application error split already documented in the capability ring and userspace runtime docs.

Recommendation

Use Pingora as precedent for a capability-native service framework: library-first, typed, phase-oriented, operationally aware, and opinionated about common mechanics.

Do not use Pingora as precedent for broad kernel interfaces, ambient service discovery, global registries, generic plugin phases, or permissive defaults. The capOS version should make authority narrower than Pingora does, because capOS has a stronger capability model available at every boundary.