Containerised build at a canonical path

make container-build compiles capOS inside a container with the worktree bind-mounted at /build. The host build path is unchanged and remains the default.

Why a container

sccache hashes the compiler process’s current working directory unconditionally for every Rust compilation, so two worktrees at different host paths never share a first-party cache entry. This is upstream-tracked and unfixed: mozilla/sccache#2652 (the Rust hash key ignores SCCACHE_BASEDIRS) and #2595 (this exact worktree case). SCCACHE_BASEDIRS, --remap-path-prefix and -Zremap-cwd-prefix were each measured at zero additional hits; the first does not apply to Rust at all despite its README, and the latter two rewrite the artifact rather than the key.

Making the directory genuinely identical is the only mechanism that works, and it is the only one that stays honest: the artifacts really do carry /build, rather than carrying whichever worktree happened to compile them first.

What must match for a cache hit

The toolchain mount path belongs in the left column for the same reason the worktree path does: cargo runs each unit’s rustc with the working directory set to that package’s own root, and a -Zbuild-std=core,alloc unit’s package root is under <toolchain>/lib/rustlib/src/rust/library/. Every userspace build alias in .cargo/config.toml uses -Zbuild-std, so parametrizing the toolchain mount would silently lose every core and alloc hit. capos-container-build.sh hardcodes /rust, which is why this is a latent trap rather than a live defect, and the ubuntu:24.04 versus python:3.12-slim spike is not counter-evidence: it held the mount at /rust in both arms.

The environment rule has two halves, and only the first is unconditional:

• Every CARGO_* variable enters the key regardless of whether anything reads it, because env!() could. An unrelated CARGO_UNRELATED=1 is enough to zero the hit rate. capos-container-build.sh therefore starts the container from an empty environment and sets the CARGO_* set explicitly, so no host variable can leak in.
• Any other variable enters the key when the compilation actually reads it. rustc records such a read as an env-dep in dep-info, and sccache tracks it. Measured with a build script emitting cargo:rustc-env and a library doing env!(): changing a non-CARGO_ variable missed, changing it back hit, and each rlib carried the value it was built with.

Reading the first half alone invites the conclusion that a non-CARGO_ variable is invisible to the key. It is not, and the difference matters here: kernel/build.rs emits CAPOS_BUILD_COMMIT / CAPOS_BUILD_TIMESTAMP / CAPOS_BUILD_VERSION as cargo:rustc-env, and kernel/src/cap/system_info.rs reads them with env!(). Treating them as invisible would be a wrong-build-identity bug: a kernel serving another commit’s identity. The observed behaviour is the safe one – see the reuse limits below.

Measured reuse

Two worktrees at the same commit, at different host paths, building make kernel (init plus kernel, so both the x86_64-unknown-capos userspace target and the x86_64-unknown-none kernel target, including the kernel link). Each arm starts from an empty cache and clean target directories, builds the first worktree to populate, then builds the second.

This table is not reproduced: it was taken once per cell, on a host running concurrent builds, and the run recorded neither the cache-isolation procedure nor the stats capture. Treat the hit and miss counts as indicative and the wall-clock figures as unreliable; the reuse ratio is the durable part, and the artifact identity below is separately reproducible. Re-measuring needs two worktrees at one commit, a dedicated SCCACHE_DIR per arm so the live host cache is neither read nor disturbed, drained target directories, and CAPOS_CONTAINER_SCCACHE_STATS=1 for the container arm (the server dies with the container, so its counters can only be read from inside).

The headline host rate is high because this goal is dominated by registry dependencies, and those already share: cargo runs each unit’s rustc with the working directory set to that package’s own root, so a crate unpacked under CARGO_HOME – or a -Zbuild-std unit compiled from the rustup rust-src component – is worktree-independent already. The 12 host misses are precisely the worktree-local first-party crates: capos-abi, capos-lib, capos-config, capos-tls, capos-rt, capos-init, capos-kernel, the vendored trees, and the build scripts. Those are the expensive units, which is why closing a 12-unit gap moved wall clock at all rather than trivially – though see the load caveat on the table above before quoting a figure.

So the rate itself is workload-dependent – a goal that builds demos/ or shell/ has a far higher first-party fraction and a correspondingly lower host baseline. The invariant underneath it is that first-party cross-worktree reuse goes from 0% to 100% over the cacheable set, at equal commits and at differing commits alike – see the different-commit measurement below, which also corrects what capos-kernel’s exclusion actually is.

The different-commit case, measured – and capos-kernel is not a miss at all

The equal-commit table above is the best case. The autopilot case is two worktrees on different branch tips, and it was left unmeasured. Measured 2026-07-26, and the result corrects the explanation this document previously gave.

Procedure, against the live shared container cache (so the numbers are the ones a real worker sees, not an isolated ideal):

Create two detached worktrees at the same mainline commit. In the second, commit one added line in a kernel source file, so its tip, its sources and its CAPOS_BUILD_COMMIT all differ from the first’s. Then, in each worktree in turn:

CAPOS_CONTAINER_SCCACHE_STATS=1 make container-build CONTAINER_GOALS=kernel

The stats variable is required: the container gets its own network namespace, so its sccache server dies with the container and its counters can only be read from inside.

First-party cross-worktree reuse at a different commit is 100% of the cacheable set: 91 of 91, zero misses. Both worktrees are at different host paths, and the second differs from the first in a kernel source file, its commit and its build identity.

The zero-miss column is the part that needed explaining, because a changed kernel source cannot possibly hit. It does not: capos-kernel is never offered to the cache. Isolated directly – touch one kernel source in the second worktree and rebuild, so the kernel is the only unit that recompiles:

Compile requests            3
Compile requests executed   0
Cache hits                  0
Cache misses                0
Non-cacheable calls         1     reason: crate-type
Non-compilation calls       2

So the earlier claim here – that CAPOS_BUILD_COMMIT in the unit’s key makes capos-kernel miss across branch tips – described the wrong mechanism. sccache declines the kernel’s rustc invocation outright on crate-type, in every build, at every commit, in both host and container mode. The environment rule is real (see above) but has no bearing on the kernel, because the kernel’s compile never reaches key computation. The consequence is a ceiling, not a regression: the container path cannot accelerate the kernel compile-and-link, and neither can the host path, so nothing is lost by switching. Of the 123 requests in this goal, 29 are non-cacheable (crate-type 15, missing input 9, stdin 5) and 3 are non-compilation probes; the reuse claim covers the other 91.

Also directly observed: two container builds of the same tree at different commits produce a byte-identical capos-init and a different kernel. With the mechanism above, that difference is simply the kernel being compiled fresh both times, not a cache decision.

The kernel linker script is not an obstacle

Worth recording, because it is easy to assume otherwise. The kernel linker script is passed as a relative path from .cargo/config.toml (-C link-arg=-Tkernel/linker-x86_64.ld), resolved against the compile working directory, which is the workspace root. It does not travel through CARGO_ENCODED_RUSTFLAGS – the Makefile only ever clears that variable, for the standalone wasm payload builds. So the kernel-target cache key carries no absolute path from the link step, and the kernel ELF links inside the container unchanged.

Layout

• tools/container-build/Dockerfile – distribution packages only: gcc plus glibc headers, make, and git. No Rust toolchain, no capOS tool cache. Pinned by base-image digest.
• tools/container-build/capos-container-build.sh – resolves the host paths and runs make <goal> inside the container. Its pre-flight refusals all come before any provisioning work: an unsupported goal, a missing container runtime, a missing rustc, a missing sccache, a missing toolchain, an image that is not present locally, a bind-mount path containing , or = (the --mount option string has no escaping for either), a container build of the same worktree already running, and a target directory in the other build mode. Ordering matters because the capnp build stamp depends on the Makefile, so any Makefile edit can otherwise make a from-source capnp build the price of being told sccache is missing. The image check matters because docker run treats a missing local image as a registry reference and reports a Docker Hub pull failure instead of “run make container-image first”. The already-running check matters because --rm only removes a container after it exits, so a SIGKILL to the CLI leaves one running; the container is named from the worktree so concurrent builds of different worktrees stay allowed, which is the normal parallel case.
• tools/container-build/capos-build-mode.sh – the build-mode guard and its self-test.

The container is not a sandbox and make container-build is not a hardening measure. $HOME/.cargo is bind-mounted read-write of necessity, since cargo unpacks registry sources into it; that directory holds credentials.toml and a bin/ that is on the host PATH, so a build script running in the container can read a crates.io token or overwrite a host binary. This is not a regression – a host build already runs the same build scripts with the same user’s rights – and nothing in this path claims isolation. It is recorded so the container is not later read as a trust boundary. --network none narrows what a build can reach, but it is there to make an unexpected fetch fail loudly, not to contain a hostile build script.

Everything version-pinned stays on the host and is bind-mounted:

• Rust toolchain – rust-toolchain.toml is already the pin, and the measured cache key depends on the compiler’s content rather than its path. Baking a toolchain into the image would add a second pin that can drift from the first, and would make image reproducibility a precondition for cache-key reproducibility. Mounted read-only at /rust.
• capnp – capos-config’s build script executes it, so it must be reachable inside the container, and tools/capnp-build asserts its exact path under CAPOS_TOOLS_ROOT. The Makefile’s existing SHA-256-pinned provisioning is unchanged; the tool cache is mounted read-only at /capos-tools and capnp-ensure inside the container only re-verifies it.
• cue, limine – reachable inside the container through the same read-only CAPOS_TOOLS_ROOT mount, still SHA-256-pinned by the Makefile. Their provisioning stays on the host: the extraction recipes write into that mount, which fails inside the container as a read-only-filesystem error from within a flock, and pinned provisioning should not silently happen in a container anyway. So the host pre-flight runs cue-ensure and limine-ensure alongside capnp-ensure, and the container only consumes them.
• xorriso – installed in the image, because unlike cue and limine it has no host provisioning to mount. It is the one image package that masters shipped artifact bytes, so it is recorded in docs/trusted-build-inputs.md rather than treated like gcc (whose output never reaches the shipped ELFs).
• mdbook, typst – docs are not compilation. They stay on the host, outside this harness.

The git common directory is mounted at its own host path, read-only, because a worktree’s .git file names that absolute path and kernel/build.rs derives the build identity (CAPOS_BUILD_COMMIT, CAPOS_BUILD_TIMESTAMP) from git. Without it the kernel would stamp itself unknown. It is not a CARGO_* variable and is not a compile working directory, so it cannot perturb the key.

Cache directory

Container and host builds compute different keys for the same crate, so they share nothing. Pointing both at one directory would only have them compete for the same SCCACHE_CACHE_SIZE bound, so the container defaults to a separate $HOME/.cache/capos-container-sccache. Override with CAPOS_CONTAINER_SCCACHE_DIR, and collapse the two once the containerised path is the only one in use. The consequence is that the host’s cache bound and the container’s are independent, so the total budget is their sum; both are stated explicitly rather than left to sccache’s default.

Stating the container’s bound explicitly is not optional tidiness. HOME inside the container is a tmpfs, so no host ~/.config/sccache/config is readable and an operator’s configured size (or any other option set there) does not apply. Without an explicit SCCACHE_CACHE_SIZE the container silently falls back to the built-in default, and this paragraph would be reasoning about a bound the container never reads.

Not covered

• Running the QEMU gates inside the container. CONTAINER_GOALS admits compilation and packaging, but not the gates themselves: they would need qemu in the image plus --device /dev/kvm and a supplementary kvm group, because QEMU_KVM_FLAGS falls back to TCG when /dev/kvm is unavailable – correct, but far too slow for the smoke timeouts. Networking needs nothing extra: the harnesses use QEMU user networking, not a TAP device. The C and wasm payload goals are also out, because their recipes drive a C compiler and a second target triple that are not verified here.
• Booting a container-built ISO from a host QEMU target – available, but only for run-smoke so far. $(ISO) depends on the compile targets, so a plain host QEMU target over a container-built tree is the mixed-mode hazard above and the guard refuses it. CAPOS_PREBUILT_ISO=1 is the way through: it drops the ISO from the target’s prerequisites and takes the UART map from the ISO’s sidecar instead of recomputing it with a host mkmanifest, so the whole flagged path compiles nothing. Only run-smoke carries the flag today. Every other QEMU target still reaches the ISO rule and is still refused in a container-mode worktree – including run-default-boot, which CLAUDE.md names as the gate for the default system.cue boot and which the container can build (all produces capos-default.iso) but not yet boot.
• Verification parity between the two packaging modes. ovmf-verify resolves OVMF_CODE by wildcard over host firmware paths, none of which exist in the image, so it degrades to its NOTICE and skips the pin check for every container-built ISO. The ISO does not embed OVMF and the firmware only matters to run-uefi, so the impact is small, but nothing records which checks the container mode skipped.
• Build-provenance identity across modes. tools/build-provenance.sh runs on the host: it reads the mode from .capos-build-mode and would therefore report Build mode: container beside the host’s xorriso version, describing a tool that did not master the artifact. The ISO’s sidecar records the mode that built it, which is the seam to wire in.

One build mode per worktree, enforced

This is a correctness rule, not a performance note, and it is enforced rather than documented. Cargo’s freshness check for a workspace member does not notice the switch: it tracks source files by a path relative to the package, and the mount makes those relative paths and their mtimes identical. So a build in one mode on top of the other mode’s target/ can mark a unit fresh and relink the rlib the other mode produced, which carries that mode’s build root in its DWARF.

Both directions reproduce, and the host-over-container direction is the worse one: the host path is the default that make run, the QEMU smokes and make build-provenance all consume. Observed there: a plain make kernel over a clean container-built target directory exited 0 with no warning and produced a kernel in which one compilation unit (capos-abi) had DW_AT_comp_dir = /build, a directory that does not exist on the host. Nothing in the build output hinted at it. The artifact looked correct.

The extent is narrower than the mechanism suggests, which matters for triaging a real occurrence: in that reproduction 92 of 93 units recompiled and exactly one crate, capos-abi, survived stale.

The reason is not simply that CARGO_HOME differs between the modes. That explains why registry units rebuild, but not why a first-party unit resolving no registry path rebuilds – capos-lib recompiled too. The operative rule is cargo’s dependency-output comparison: once a dependency’s rlib is rewritten, every transitive dependent is dirty regardless of its own dep-info. That predicts the survivor set is exactly the dependency-free first-party leaves, which is a property of the workspace graph rather than a constant. capos-abi is currently the only workspace member with no dependencies and no build-dependencies, which is why it was the only survivor. If it ever gains a dependency, nothing survives a mode switch and the hazard disappears by accident; if a second dependency-free crate is added, two survive.

The mechanism is inferred from cargo’s fingerprint model rather than measured – what was measured is the outcome, 92 of 93 – but the graph premise is checkable against the workspace manifests, and the practical consequence stands either way: one mixed unit is enough to make the artifact wrong, and a mostly-reused build is not what an occurrence looks like.

tools/container-build/capos-build-mode.sh closes this. It records the mode that last wrote build output in .capos-build-mode (gitignored, repository root) and refuses a build in the other mode while any target directory still holds output. capos-container-build.sh additionally runs it before docker run so a refusal costs nothing. make container-build-mode-test is its self-test.

Its reach into the Makefile is the assumption everything else here rests on, so it is asserted rather than described. The guard is a prerequisite of capnp-ensure, which covers every target that compiles capOS code for a capOS target — but that is not the same as every cargo invocation, and the difference was larger than a reading of the Makefile suggested.

Writing the exception list by hand produced six targets. Computing the prerequisite closure from make’s own rule database found sixteen cargo-invoking rules that did not reach the guard. Three families mattered:

• model-scheduler-nohz-loom, a host-triple capos-config Loom test, and wasi-env-negative-check, a host-triple mkmanifest test, both write the root target/ — the directory a container build fills. An unguarded write there can rewrite target/.rustc_info.json with the host sysroot inside a directory holding container output, which is the evidence the unmarked-worktree inference below depends on.
• Eleven demos/wasi-*/target/wasm32-wasip1/release/*.wasm artifact rules, reachable through their wasi-*-build parents. These compile wasm32-wasip1, which disproves the reasoning offered for the hand list — that every bypass was host-triple and therefore harmless. Their directories are already in the guard’s discovery set, so the guard could refuse a build over output nothing had declared.
• capos-tls-test (deliberately capnp-free, since it builds only -p capos-tls), webui-login-peer-logic-test, userspace-broker-policy-test, task-coordinator-logic-test, fuzz-build and fuzz-smoke. The first two are make check members, so without the prerequisite make check in a container-mode worktree compiled into demos/target/ before failing at capos-rt-check.

All of them now reach the guard; the eleven artifact rules take it as an order-only prerequisite so a .PHONY dependency does not make every .wasm perpetually out of date.

Only three cargo-invoking rules are deliberately unguarded, and tools/container-build/check-cargo-guard-coverage.py asserts set equality against exactly that allowlist: fmt and fmt-check compile nothing, and clean is the remedy for a refusal, so guarding it would strand a worktree that could then neither build in the recorded mode nor drain out of it. A newly unguarded rule fails the gate, and so does an allowlisted rule that becomes guarded. Widening the allowlist is a reviewed diff to that file.

Two checks run, deliberately: the closure computation above is complete but parses make -p output, so it also asserts a floor on the number of cargo rules it found — a degraded parse would otherwise report everything as guarded. Beside it, capos-build-mode.sh --check-guarded-targets asks make --dry-run what a sample of targets would actually run. One is complete, the other is authoritative.

Three properties are worth stating because they are what make it fail closed:

• Target directories are discovered, not listed. Any directory named target up to three levels deep that holds build output counts. This is the property that makes the guard cover the whole goal set rather than the default goal: nine standalone crate manifests build through .cargo/config.toml aliases – init/, demos/, shell/, capos-rt/, capos-service/, capos/, libcapos/, libcapos-posix/, capos-wasm/ – each with its own <crate>/target, and the demos/wasi-* payloads are separate workspaces one level deeper. CONTAINER_GOALS is overridable, so shell, demos, capos-rt-check and friends are ordinary usage, and by the reuse argument above they are the goals with the highest first-party fraction – the ones this path pays best on. A target plus init/target list would leave every one of them exposed. make container-build-mode-test asserts the whole set: restricting discovery to those two directories fails the gate and names the eleven it missed.
• An unmarked worktree is classified from evidence. Cargo records the rustc sysroot in <target-dir>/.rustc_info.json, and the container’s sysroot is the fixed mount path /rust, so an unmarked directory still declares which mode filled it. Without this, a worktree holding container output produced before the guard existed would be misread as host and permitted – exactly the mix the guard is for. Every occupied directory is classified rather than the first one that answers, so a tree that is already mixed is reported as such and refuses both modes instead of being adopted as whichever mode was found first.
• It refuses rather than wiping. Discarding build output is the operator’s call; the refusal names the directories and prints the command.

Two known limits, neither reachable from a container goal today. Discovery is complete to a hardcoded depth of three rather than complete by construction; tools/remote-session-client/src-tauri/target sits at depth four, and no allowlisted goal writes it. And the guard checks state without reserving it, so a host make starting inside a container build’s startup window is accepted and flips the marker. Closing the second needs a lock, not a deeper glob.

The cost is that the modes are exclusive for the whole worktree, not per goal. After a full host ISO build, switching to container builds means discarding every populated target directory, and vice versa. That is the same “pick one mode per worktree” rule as before, now with a build-time signal instead of a convention. Note target/.gitkeep is tracked, so rm -rf target needs it restored afterwards.

Consequently a container-mode worktree cannot run the host gates (make check and everything under it reaches capnp-ensure), and a host-mode worktree cannot run make container-build without draining first. Drain a worktree – empty target directories and no marker – to leave it free for either mode.

Artifacts

Two worktrees at different host paths, each built clean in a container, produce a byte-identical kernel ELF (same SHA-256) and a byte-identical capos-init ELF. The kernel contains no host path at all: every first-party compilation unit’s DW_AT_comp_dir is /build, and so are the embedded source-path strings. A host build of the same tree carries the worktree path in both.

Measured on the 232e1791 container-built kernel:

K=target/x86_64-unknown-none/debug/capos-kernel
sha256sum "$K" init/target/x86_64-unknown-capos/release/capos-init
readelf --debug-dump=info "$K" | grep -c 'DW_AT_comp_dir.*: /build$'   # 49
readelf --debug-dump=info "$K" | grep DW_AT_comp_dir | sed 's/.*: //' \
    | sort -u | wc -l                                                  # 22
strings -a "$K" | grep -oE '^/build/[A-Za-z0-9._/-]+' | sort -u         # 5 paths
strings -a "$K" | grep -c '/home/'                                     # 0

The 49 are the first-party units; the 22 distinct comp_dir values are those plus the /cargo/registry/... and /rustc/<hash> roots, which are worktree-independent in either mode. The five embedded source-path strings are capos-rt/src/{panic,lib,entry}.rs, capos-config/src/validation.rs, and the generated capos_capnp.rs under the build-script OUT_DIR. All five counts are commit-specific – the OUT_DIR one embeds a build hash – so re-measure rather than quoting them.

That is what makes serving one worktree’s cached object to another sound rather than a fudge: the compiler really did run in /build both times. Key normalisation alone would leave the artifact carrying whichever build root compiled it first, which is unacceptable for a kernel that prints paths.

Remaining caveats

• DW_AT_comp_dir becomes /build. Host-side debuggers resolving source from container-built artifacts need a path mapping (set substitute-path /build <worktree> in gdb).
• Panic locations change for the standalone crates. Workspace members are unaffected: cargo passes relative source paths for them, so their file!() strings are identical either way. init/ and its outside-the-package path dependencies are not, because cargo passes those as absolute paths. The container-built kernel, which embeds capos-init via include_bytes!, carries /build/capos-rt/src/panic.rs, /build/capos-rt/src/lib.rs, /build/capos-rt/src/entry.rs and /build/capos-config/src/validation.rs where a host build of the same tree carries the worktree path. So init-side panic output names a directory that does not exist on the host.
• The total sccache budget doubles. The container defaults to its own cache directory (see above), and neither cache is a subset of the other, so the bound is the host’s plus the container’s – currently 10G each. Both are set explicitly (SCCACHE_CACHE_SIZE, and CAPOS_CONTAINER_SCCACHE_CACHE_SIZE for the container) rather than inherited from sccache’s default. A server that is already running keeps the size it started with.
• Build mode is recorded in provenance, and the two are not comparable. make build-provenance reports Build mode and Build container image from the build-mode marker, because every other identity field in that record describes the host while the artifact bytes may not. A host artifact and a container artifact differ by construction, so a comparison across modes is not a reproducibility result; see the retention and comparison policy in docs/trusted-build-inputs.md.
• The image is a build input. It is pinned by digest and inventoried in docs/trusted-build-inputs.md, which also carries the refresh procedure.