# Self-Hosted Task Backend

Detailed decomposition for hosting the multi-agent development task backend on
capOS itself. Today that backend is loopyard, a companion host-side project
(PostgreSQL-backed; not part of this repository): it computes runnable task
sets from typed dependencies, validates status transitions, checks conflict
domains, and issues lock leases with generation fencing, all enforced inside
the database so no client can corrupt task state, with CLI/MCP/REST as thin
transports over one API surface. The long-term intent — recorded in
[Stateful Task and Job Graphs](../proposals/stateful-task-job-graphs-proposal.md)
and [capOS-Hosted Agent Swarms](../proposals/hosted-agent-swarm-proposal.md) —
is for capOS to serve that coordination role in-system through typed
capability services instead of an external SQL server.

## Visible Outcome

A capOS instance serves the task backend for the multi-agent development
workflow that builds capOS: host-side workers resolve runnable tasks, acquire
fenced lock leases, report status transitions, and receive conflict reports
from a capOS-hosted coordinator service, speaking the same task-source and
lock wire contracts that loopyard serves today. loopyard's schema and
invariants are the reference semantics the capOS services must reproduce, and
task state must be exportable/importable in both directions for migration.

## Gap Inventory

What exists and what is missing, largest gap first:

1. **Transactional structured record store.** The coordinator needs
   record-level atomic multi-record commits, secondary lookups, and
   constraint-style invariant enforcement. Current durable storage is below
   that level: the userspace `Store`/`Namespace` service persists a
   whole-state snapshot (`CAPOSUS1`), the kernel `persistent_store` fixture is
   a disk-backed `CAPOSST1` `Store`, and `writable_fs` is a single-writer
   `CAPOSWF1` filesystem. Crash-safe A/B superblock commits and torn-grow
   mount recovery are landed building blocks one level down. Missing: a
   write-ahead-log or equivalent record store over `BlockDevice` with atomic
   multi-record transactions.
2. **The coordinator service itself.** A userspace service owning task
   records, statuses-as-data with validated transitions, dependency edges
   with cycle rejection, runnable-set computation, conflict-domain checks,
   and lock leases with expiry and generation fencing — the
   [Stateful Task and Job Graphs](../proposals/stateful-task-job-graphs-proposal.md)
   Stage E operator task surface and Stage G agent workflows, with invariants
   enforced in the service the way loopyard enforces them in PL/pgSQL.
   Nothing is implemented.
3. **Public ingress and TLS.** Host-side workers need to reach the backend.
   Private GCE self-hosted Web UI serving is proven; public exposure and TLS
   remain gated by the explicit on-hold
   [`cloud-gce-public-self-hosted-webui-ingress-tls`](../tasks/on-hold/cloud-gce-public-self-hosted-webui-ingress-tls.md)
   task. TLS building blocks (`capos-tls`, the ACME http-01 solver) exist.
4. **A reusable multi-client API service layer.** The remote-session Web UI
   already runs a persistent accept/recv/send/close loop with per-connection
   deadlines and slow-client bounds over the Phase C userspace stack, but as
   a demo-grade single flow. The backend needs a reusable HTTP/JSON service
   layer (a natural consumer for the `capos-service` lifecycle crate, which
   currently has none in-tree) plus readiness-driven multiplexing of
   concurrent client connections over the landed poll/select readiness
   bridge.
5. **Remote actor authentication.** loopyard's principal/agent-session actor
   model maps naturally onto `user_session`, `credential_store`, and
   `authority_broker`, but API-token issuance and per-request actor
   attribution for programmatic clients do not exist; the Web UI login is
   demo-grade.
6. **Client transport.** Either a REST/JSON adapter (matching loopyard's
   thin-transport shape; fastest for the vibe-loop command contracts) or the
   `capos` SDK `remote` transport
   ([capOS SDK And Dual Transport](capos-sdk-dual-transport.md)); the
   transitional host-backend remote transport (slice 4a there) can carry
   typed clients before the full `capnp-rpc` upgrade.
7. **Time quality and data portability.** `WallClock` serves UTC over a fixed
   boot base, so lease expiry and timestamps are not comparable across
   reboots without time synchronization. Export/import tooling is needed for
   migration to and from the host backend and for backups.

## Sequencing

Phases ordered so each lands on local QEMU evidence before touching the gaps
behind it:

1. **In-memory coordinator local proof** (dispatchable now; no dependency on
   ingress or storage): coordinator service plus client demo proving
   runnable-set, transition validation, conflict domains, and fenced leases
   in QEMU. Root task record: `task-backend-coordinator-inmemory-local-proof`.
2. **API surface local proof**: HTTP/JSON adapter over the userspace network
   stack exposing the coordinator to a host-side client speaking the
   task-source/lock command contracts against local QEMU.
3. **Durable record store**: the transactional record store over
   `BlockDevice`, then switch the coordinator's persistence from in-memory
   state to it, with reboot and crash-recovery proofs.
4. **Actors, time, and portability**: API-token authentication bound to the
   session/credential surface, time synchronization for cross-reboot lease
   and timestamp comparability, and export/import against the host backend's
   data.
5. **Live deployment**: serve the backend behind the public ingress/TLS
   milestone once that separate track is authorized and closed.

Phases 1 and 2 can proceed against in-memory state while phase 3 is designed;
the coordinator's storage seam should keep the record-store swap mechanical.

## Design Grounding

- [Stateful Task and Job Graphs](../proposals/stateful-task-job-graphs-proposal.md)
  — the work-graph substrate this track exercises (Stages E and G) and the
  prior-art analysis, including loopyard.
- [capOS-Hosted Agent Swarms](../proposals/hosted-agent-swarm-proposal.md) —
  `AgentTask`, `ResourceLease`, `ResourceVersion`, and `ConflictReport`
  shapes the coordinator should converge toward.
- [Task State and Agent Telemetry](../proposals/task-state-and-agent-telemetry-proposal.md)
  — the current file-per-task ledger the development workflow uses inside
  this repository.
- [capOS SDK And Dual Transport](capos-sdk-dual-transport.md) — the remote
  transport option for typed clients.