Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

AWS Nitro EBS (NVMe storage)

This is a provenance map for the AWS Nitro EBS storage shape: how an AWS Nitro instance presents its EBS volumes to the guest, why that surface is the same standard NVMe device the shared NVMe storage-provider foundation already drives, and the small AWS delta capOS adds on top of it. It is not a re-spec; the NVMe register/queue/PRP wire subset capOS actually touches is documented once in NVMe and not repeated here.

Maturity caveat. This page documents a local QEMU cloud-shape classification, not a bound driver running on real AWS hardware. The NVMe bind/identify/read lifecycle is proven locally on make run-pci-nvme against QEMU’s -device nvme; the AWS delta is the AWS-context classification proof line and the Nitro DMA-backend policy note on top of that shared NVMe foundation. End-to-end AWS EBS enumeration, live namespace I/O, and cloud evidence capture are future work (tracked as cloud-aws-storage-live-proof), blocked until AWS access is provisioned. The ENA NIC is a distinct driver-binding claim (cloud-aws-ena-nic-live-proof) and is out of scope here.

1. Spec basis

  • Device: AWS Nitro EBS controller. All AWS Nitro-based instance families (effectively all current generations) expose attached EBS volumes as NVMe namespaces behind a standard NVMe PCI controller – there is no AWS-specific storage transport and no virtio-scsi alternative (unlike GCP, whose first-/second-generation families use virtio-scsi). PCI class 0x01 (mass storage), subclass 0x08 (NVM), programming interface 0x02 (NVM Express) – the same class triple QEMU emulates with -device nvme and the kernel detects with PciDevice::is_nvme_controller (kernel/src/pci.rs).
  • Production PCI identity: the Nitro EBS controller carries Amazon’s PCI vendor id 0x1d0f (device id 0x8061 for the EBS NVMe controller), distinct from QEMU’s 0x1b36. capOS therefore classifies on the device class surface and the brokered no-IOMMU bounce DMA shape, not on a vendor-id match (see §3); the live vendor-id confirmation belongs to the deferred cloud-aws-storage-live-proof.
  • Authoritative spec: the NVM Express Base Specification (NVMe 1.4 / 2.0) is the wire contract; AWS publishes no separate EBS register spec because the device is a standard NVMe controller. AWS documents the namespace exposure in the “Amazon EBS and NVMe on Linux instances” guide (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html); the in-guest reference driver is the upstream Linux drivers/nvme/host/.
  • Wire-format subset capOS implements: identical to the standard NVMe subset documented in NVMe §1-§2 (controller registers CAP/CC/ AQA/ASQ/ACQ/CSTS, the admin and one I/O submission/completion queue pair, per-queue doorbells, and PRP1/PRP2 data pointers). Nitro EBS adds no fields beyond that subset, so this page does not re-list them.

2. Wire format (relevant subset)

See NVMe §2 and §6-§8. There is no AWS-specific wire format to document: the brokered controller enable (manager-authored AQA/ASQ/ACQ), the admin IDENTIFY, the one I/O queue pair, and the bounded READ all use the standard NVMe encoding the shared foundation already implements and proves.

3. capOS mapping

The AWS delta is a cloud-shape classification plus a DMA-backend policy consumption detail layered onto the shared NVMe storage-provider foundation; it adds no new driver code.

  • Cloud-shape classification proof: after the first enumerated NVMe controller is bound (bind_qemu_nvme_controller), the enumeration path emits a nvme: cloud shape classification cloud_shape=aws-nitro-ebs ... proof line (kernel/src/pci.rs report_cloud_nvme_shape) classifying the bound controller against the documented AWS Nitro EBS device surface. It prints the enumerated pci_vendor/pci_device_id and class/subclass/prog_if, records the production aws_nitro_ebs_vendor=0x1d0f identity as documentation (not as a claimed match), and carries explicit scope flags (local_qemu_precursor=true, real_aws_enumeration=not-claimed, ena=separate-nic-driver-out-of-scope). make run-pci-nvme asserts this line conjunctively with the bounce-buffer dma: backend selection line (tools/qemu-pci-nvme-smoke.sh assert_nvme_cloud_shape), tying the bound device surface to the DMA backend resolved that boot.
  • Nitro IOMMU-availability DMA-backend policy: AWS Nitro does not guarantee guest VT-d remapping the way QEMU’s emulated IOMMU does, so the DMA backend the live AWS path consumes is selected by cloud-dma-backend-selection (kernel/src/dma_backend.rs select_and_report): direct-remapping where a usable+safe IOMMU is positively probe-verified, else the labeled bounce-buffer fallback. The classification line labels the expected backend (aws_labeled_dma_backend=bounce-buffer, dma_backend_policy=direct-remapping-if-verified-else-bounce-buffer); the resolved backend is proven separately by the dma: backend selection line, which on the no-IOMMU make run-pci-nvme gate is bounce-buffer.
  • Brokered DMA / no host-physical exposure: the binding lifecycle reuses the brokered no-IOMMU lane documented in NVMe §6-§8 – the manager authors every address-bearing register and PRP from the live DMA ledger, and host_physical_user_visible=false holds throughout. On a verified remapping lane the provider-written Model B path would apply instead; on the no-IOMMU gate the brokered bounce shape is the only consistent path (see docs/dma-isolation-design.md, “Provider-Written Addresses And No-IOMMU Brokered Bounce”).
  • DeviceMmio / Interrupt / DMAPool: unchanged from the shared foundation – the reset-only CC selected-write claim, the brokered admin and I/O doorbells, the interrupt-driven admin completion wake, and the DMAPool-allocated queue/data pages described in NVMe §4-§8.
  • QEMU-emulable vs hardware-only: the classification and the full bind/identify/read lifecycle are end-to-end QEMU-emulable (make run-pci-nvme). Live EBS enumeration over a real Nitro controller – vendor-id 0x1d0f confirmation, real namespace geometry, and live block I/O – is hardware-only and is the deferred cloud-aws-storage-live-proof.
  • NVMe – the shared NVMe controller wire subset and brokered no-IOMMU storage-provider foundation this shape binds onto.
  • virtio-net – the worked cloud-shape classification example (GCP virtio-net) this page mirrors for AWS storage.
  • docs/dma-isolation-design.md – the DMA-backend selection model and the no-IOMMU brokered bounce policy.
  • docs/backlog/hardware-boot-storage.md – the cloud device tracks, including the deferred live-AWS storage proof.