AWS AWSEC2: Rewrite Nitro Sandboxes guide; add attestation and IMDS exposure guidance
Summary
Major restructure of the Amazon EC2 Nitro Sandboxes topic: reorganized table of contents and sections (How it works, Confidential and direct modes, When to use, Supported features, Cryptographic attestation, Considerations and limitations), added a dedicated Cryptographic attestation section describing NSM-signed measurements and attestation-gated AWS KMS key policies (including the debug-mode caveat), replaced the launch/resize/storage walkthroughs with links to Firecracker and attestation topics, added a warning that a Nitro Sandbox can reach the parent instance's IMDS and read its IAM role credentials, expanded considerations (memory pools/slots, device limits, lifecycle/hibernation, no GPU passthrough, nested virtualization and Credential Guard incompatibility), and updated supported instance type tables.
Security assessment
The added text documents a concrete weakness in which a Nitro Sandbox can reach the parent instance's IMDS and steal its IAM role credentials, notes that IMDSv2 and the hop limit of 2 do not prevent it, and gives an explicit mitigation (blocking 169.254.169.254 from Nitro Sandboxes). Alongside this, the change adds security documentation for cryptographic attestation and attestation-gated AWS KMS key policies, plus warnings that EBS/SSE-KMS encryption and parent-instance memory access do not protect data from the parent operator.
Evidence
+A Nitro Sandbox has no [instance metadata service (IMDS)](./ec2-instance-metadata.html) of its own. If the parent instance routes traffic from a Nitro Sandbox, for example with NAT, the Nitro Sandbox can reach the IMDS of the parent instance and read its AWS Identity and Access Management (IAM) role credentials. IMDSv2 and the hop limit of 2 that Amazon Linux 2023 AMIs use do not prevent this. To block this access, reject traffic from Nitro Sandboxes to `169.254.169.254` on the parent instance.
Diff
diff --git a/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.md b/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.md index 9335308..2f305e6 100644 --- a//AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.md +++ b//AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.md @@ -7 +7 @@ -How it worksConfidential and direct modesSupported featuresWhen to use Nitro SandboxesRun a Nitro Sandbox on an Amazon EC2 instanceLaunch, resize, and attach storage to a Nitro SandboxConsiderations and limitations +How it worksConfidential and direct modesWhen to use Nitro SandboxesSupported featuresCryptographic attestationConsiderations and limitations @@ -11 +11 @@ How it worksConfidential and direct modesSupported featuresWhen to use Nitro San -An _AWS Nitro Sandbox_ is a lightweight virtual machine, or child VM, that you can run alongside your workload on almost any Amazon EC2 instance. Nitro Sandboxes are built on the same AWS Nitro technology that isolates every Amazon EC2 instance. You don't rely on the guest operating system to hold the boundary. Instead, you trust Nitro, a minimized hypervisor that AWS maintains and patches. Nitro isolates each Nitro Sandbox from the parent instance, from other Nitro Sandboxes, and from AWS operators. As a result, you can isolate a workload even from your own operators, meaning the people and software that administer the parent instance. The parent contributes vCPU and memory. Nitro keeps the workload isolated and can attest to the code it runs. +AWS Nitro Sandboxes is an Amazon EC2 feature that runs a workload in its own lightweight virtual machine, called a _Nitro Sandbox_. A Nitro Sandbox runs alongside an Amazon EC2 instance, called the _parent instance_ , on vCPUs and memory that the parent instance contributes. A virtual machine monitor (VMM) on the parent instance manages it, and the AWS Nitro System enforces the boundary around it. To use Nitro Sandboxes, you must enable AWS Nitro Enclaves when you launch the parent instance. @@ -13 +13 @@ An _AWS Nitro Sandbox_ is a lightweight virtual machine, or child VM, that you c -Nitro Sandboxes build on the same foundations as AWS Nitro Enclaves, but they provide the parent instance with mechanisms to implement a wide variety of virtual hardware, such as virtual network and virtual block devices. The parent instance can use them to proxy Elastic Network Adapter (ENA) interfaces or Amazon EBS volumes into the Nitro Sandbox. As a result, a Nitro Sandbox can use ordinary networking and block storage. Standard virtual machine monitors (VMMs) manage it from the parent instance. Nitro Sandboxes run on Intel, AMD, and AWS Graviton processors. +You can use Nitro Sandboxes in the following ways: @@ -15 +15 @@ Nitro Sandboxes build on the same foundations as AWS Nitro Enclaves, but they pr -Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance and need their own virtual machine boundary, such as agentic workloads that run untrusted code, multi-tenant containers, Kubernetes pods, and Amazon ECS tasks. You should use [AWS Nitro Enclaves](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html) for small, tightly scoped jobs, because an enclave connects to its parent instance through a single local vsock channel that keeps the exposed surface small. [Nested virtualization](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nested-virtualization.html) is best for workloads that must run their own hypervisor, like Windows Subsystem for Linux, and you want full control of that hypervisor. If you want to run workloads that stay isolated between tenants, and don't want to manage servers or networking, use [AWS Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html) to offload your infrastructure management. For guidance on choosing Nitro Sandboxes for a given workload, see When to use Nitro Sandboxes below. + * For containers on Amazon Elastic Kubernetes Service (Amazon EKS), the service runs each pod in its own Nitro Sandbox. For more information, see [Sandboxing on Amazon EKS with isolated and confidential pods](https://docs.aws.amazon.com/eks/latest/userguide/sandboxing.html) in the _Amazon EKS User Guide_. @@ -17,7 +17 @@ Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance -###### Contents - - * How it works - - * Confidential and direct modes - - * Supported features + * For containers on Amazon Elastic Container Service (Amazon ECS), the service runs each task in its own Nitro Sandbox. For more information, see [Isolated and confidential tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/isolated-confidential-tasks.html) in the _Amazon Elastic Container Service Developer Guide_. @@ -25,3 +19 @@ Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance - * When to use Nitro Sandboxes - - * Run a Nitro Sandbox on an Amazon EC2 instance + * With your own orchestrator on an Amazon EC2 instance, you launch Nitro Sandboxes with Firecracker. For more information, see [Run Nitro Sandboxes with Firecracker](./nitro-sandboxes-run-firecracker.html). @@ -29,3 +20,0 @@ Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance - * Launch, resize, and attach storage to a Nitro Sandbox - - * Considerations and limitations @@ -34,0 +24 @@ Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance +For confidential workloads, you verify the attestation of each Nitro Sandbox with an AWS Key Management Service (AWS KMS) key policy, whichever way you launch it. @@ -36,5 +26 @@ Nitro Sandboxes suit a wide range of workloads that share an Amazon EC2 instance -## How it works - -A Nitro Sandbox runs as a first-level, hardware-virtualized guest alongside your instance. The parent instance provides the Nitro Sandbox with vCPU, memory, and I/O, while the Nitro System enforces the isolation boundary and provides cryptographic attestation. Nitro Sandboxes are isolated from each other and from the parent instance. No matter what runs inside a Nitro Sandbox, it can't access another Nitro Sandbox or the parent. - -You select a mode (confidential or direct) and a VMM when you create a Nitro Sandbox. For the difference between the modes, see Confidential and direct modes. For the VMM options and how Firecracker drives a Nitro Sandbox, see How Firecracker works. +There are no additional charges for using Nitro Sandboxes. Nitro Sandboxes run on the vCPUs and memory of the parent instance, so you pay the standard charges for the Amazon EC2 instance and for the other AWS services that you use. @@ -42 +28 @@ You select a mode (confidential or direct) and a VMM when you create a Nitro San -## Confidential and direct modes +###### Contents @@ -44 +30 @@ You select a mode (confidential or direct) and a VMM when you create a Nitro San -Every Nitro Sandbox runs in one of two modes. You choose the mode when you create a Nitro Sandbox, and you can't change it while the Nitro Sandbox is running: + * How it works @@ -46 +32 @@ Every Nitro Sandbox runs in one of two modes. You choose the mode when you creat -Confidential mode (default) + * Confidential and direct modes @@ -47,0 +34 @@ Confidential mode (default) + * When to use Nitro Sandboxes @@ -49 +36 @@ Confidential mode (default) -The Nitro System moves the Nitro Sandbox's memory out of the parent instance, so the parent can't read it, and the Nitro Sandbox can produce an attestation of the code it is running. The Nitro Sandbox includes a Nitro Security Module (NSM) for attestation, and virtio device queues use dedicated shared memory regions so that devices work without exposing the rest of guest memory. Use confidential mode for any workload that processes secrets, including protection from your own operators. + * Supported features @@ -51 +38 @@ The Nitro System moves the Nitro Sandbox's memory out of the parent instance, so -In confidential mode, a workload can prove that it is running unmodified code before it receives secrets. You verify this attestation with an external service, commonly AWS KMS, by adding a condition to an AWS KMS key policy so that the service releases secrets only to a Nitro Sandbox whose measurements match the values you expect. + * Cryptographic attestation @@ -53 +40 @@ In confidential mode, a workload can prove that it is running unmodified code be -Direct mode + * Considerations and limitations @@ -54,0 +42 @@ Direct mode + * [Run Nitro Sandboxes with Firecracker](./nitro-sandboxes-run-firecracker.html) @@ -56 +44 @@ Direct mode -The parent shares the Nitro Sandbox's memory instead of taking exclusive ownership of it. There is no attestation, and the parent retains access to the Nitro Sandbox's memory. Direct mode is simpler and offers higher I/O throughput, although it is not confidential. Its trust model is equivalent to a standard nested virtual machine. Use direct mode when the Nitro Sandbox and the parent are operated by the same trusted party. + * [Attest a Nitro Sandbox](./nitro-sandboxes-attestation.html) @@ -58 +45,0 @@ The parent shares the Nitro Sandbox's memory instead of taking exclusive ownersh -## Supported features @@ -60 +46,0 @@ The parent shares the Nitro Sandbox's memory instead of taking exclusive ownersh -A Nitro Sandbox presents a standard virtio-pci device model to the guest, so your workload runs without sandbox-specific code. The following features are available in a Nitro Sandbox: @@ -62 +47,0 @@ A Nitro Sandbox presents a standard virtio-pci device model to the guest, so you -Block storage @@ -63,0 +49 @@ Block storage +## How it works @@ -65 +51 @@ Block storage -Attach block devices (virtio-blk) backed by local instance store or Amazon EBS. You can resize a block device while the Nitro Sandbox is running. A block device attaches to only one Nitro Sandbox at a time. +A Nitro Sandbox runs on the Nitro Hypervisor alongside its parent instance, the same way that an Amazon EC2 instance does, so its vCPUs run at Amazon EC2 instance level performance. The Nitro System isolates Nitro Sandboxes from each other, and code in a Nitro Sandbox can't access the memory of the parent instance. A Nitro Sandbox reaches the parent instance only through its virtual devices. @@ -67 +53 @@ Attach block devices (virtio-blk) backed by local instance store or Amazon EBS. -Networking +The VMM creates Nitro Sandboxes through the Nitro Enclaves driver on the parent instance and implements their virtual devices. Currently, Firecracker is the only VMM that supports Nitro Sandboxes, and the driver is available for Linux. For the supported versions, see [Prerequisites](./nitro-sandboxes-run-firecracker.html#nitro-sandboxes-prerequisites). @@ -68,0 +55 @@ Networking +## Confidential and direct modes @@ -70 +57 @@ Networking -Attach network interfaces (virtio-net) for VPC connectivity, and reach instance metadata through the instance metadata service. Each Nitro Sandbox has its own isolated network stack and can't share the parent instance's network namespace. +For each Nitro Sandbox, you choose how much the parent instance can observe. You choose the mode at launch, and you can't change it later. @@ -72 +59 @@ Attach network interfaces (virtio-net) for VPC connectivity, and reach instance -Host communication +Confidential mode (default) @@ -75 +62 @@ Host communication -Use a vsock channel (virtio-vsock) between the Nitro Sandbox and the parent, for example to relay attestation traffic. +The Nitro System removes the memory of the Nitro Sandbox from the parent instance, so neither the parent instance nor its administrators can read or change that memory. Each virtual device exchanges data through its own dedicated buffer. A Nitro Security Module (NSM) in the Nitro Sandbox provides cryptographic attestation. @@ -77 +64 @@ Use a vsock channel (virtio-vsock) between the Nitro Sandbox and the parent, for -Entropy +The parent instance still carries the device traffic, chooses the image, and controls when the Nitro Sandbox runs. Encrypt data that leaves the Nitro Sandbox, and use attestation to check its image before you give it secrets. @@ -78,0 +66 @@ Entropy +Use confidential mode for workloads that you keep private from the administrators of the parent instance. @@ -80 +68 @@ Entropy -Attach an entropy device (virtio-rng) in direct mode. Confidential mode doesn't support the entropy device. The guest uses its own entropy sources instead. +Direct mode @@ -82 +69,0 @@ Attach an entropy device (virtio-rng) in direct mode. Confidential mode doesn't -Device hot plug @@ -83,0 +71 @@ Device hot plug +The parent instance shares the memory of the Nitro Sandbox and can read and write it. There is no NSM and no attestation. The virtual devices exchange data directly through the memory of the Nitro Sandbox, without dedicated buffers, and the trust model is equivalent to that of a nested virtual machine. @@ -85 +73 @@ Device hot plug -Add block and network virtio devices to a running Nitro Sandbox when you enable hot plug. The devices that you hot plug count toward the instance's device limit. You can't hot-unplug a device. +Use direct mode when the same trusted party operates both the Nitro Sandbox and the parent instance. @@ -89,49 +77 @@ Add block and network virtio devices to a running Nitro Sandbox when you enable -Use a Nitro Sandbox when you need a boundary as strong as a virtual machine around a container-scale workload, but you don't want the cost of a separate Amazon EC2 instance for each workload or the overhead of nested virtualization. Common use cases include: - - * **Enhanced container isolation** – Make each pod or task a hard security boundary in multi-tenant clusters, while still packing many workloads onto each instance. - - * **Attested containers** – Let regulated workloads or third-party models prove their isolation and integrity before they receive secrets. - - * **Attested virtual machine workloads** – Use a Nitro Sandbox as an alternative to nested virtualization for lightweight virtual machines. It is available on all instance types that support Nitro Sandboxes and provides higher performance together with hardware-backed attestation. - - * **Isolation from your own operators** – Run a workload so that neither the parent instance nor your own operators can read its memory or data. - - * **Secret isolation** – Isolate secrets and sensitive logic from a larger application, so that a compromise of the application doesn't expose them. - - * **Agentic workloads** – Sandbox AI agents that generate and run untrusted code, giving each agent session a hardware-isolated boundary. - - - - -###### Note - -You can use Nitro Sandboxes directly on Amazon EC2, or through a managed container service. If you run containers on Amazon EKS or Amazon ECS, then Amazon EKS and Amazon ECS provisions and attests a Nitro Sandbox for each pod or task for you. For more information, see [Sandboxing](https://docs.aws.amazon.com/eks/latest/userguide/sandboxing.html) in the _Amazon EKS User Guide_ and [Isolated and confidential tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/isolated-confidential-tasks.html) in the _Amazon ECS Developer Guide_. The rest of this topic describes the self-managed experience, in which your own orchestrator drives Nitro Sandboxes directly on Amazon EC2. - -## Run a Nitro Sandbox on an Amazon EC2 instance - -You can use Nitro Sandboxes as a building block for Amazon EC2 workloads. The managed container integrations for Amazon EKS and Amazon ECS are layers built on top of this primitive. If you run your own orchestrator, it performs the same role that a managed service otherwise would: On each instance it launches Nitro Sandboxes, connects their I/O, accounts for the resources they consume, and relays attestation. The self-managed workflow has the following main steps. - -###### To run a workload in a self-managed Nitro Sandbox - - 1. **Launch a supported instance and enable the feature.** Launch an instance type that supports AWS Nitro Enclaves, and enable it with the same instance option that enables Nitro Enclaves. For the instance launch parameters, see the [run-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) command. - - 2. **Build a guest image for your workload.** The image declares the capabilities it requires, such as the devices to attach and whether Nitro Sandboxes share CPU. These settings are part of the signed image and must match what you request at launch, which is what keeps attestation meaningful. - - 3. **Launch the Nitro Sandbox with a VMM.** On the instance, use a standard VMM (we recommend Firecracker) to launch the Nitro Sandbox from your image, specifying the vCPU count, memory, and devices. Confidential mode is the default. You can instead select direct mode for higher throughput without attestation. - - 4. **Attest the Nitro Sandbox before you release secrets.** At launch, the Nitro Sandbox measures its contents into the NSM. Your workload retrieves the attestation, and an external verifier (commonly AWS KMS, through a condition in a key policy) releases secrets only when the measurement matches. The parent instance can't forge the measurement. - - - - -When you run your own orchestrator, you are the parent operator. You are responsible for the following tasks: - - * Building guest images with the correct settings. - - * Running the device backends, and treating all data from a Nitro Sandbox as untrusted input. - - * Accounting for the memory and CPU you contribute, and zeroing memory before you contribute it. - - * Terminating Nitro Sandboxes for an orderly shutdown. - - * Relaying attestation traffic between the Nitro Sandbox and the external verifier. +Choose by how much of the machinery you want to run yourself: @@ -138,0 +79 @@ When you run your own orchestrator, you are the parent operator. You are respons + * None of it: use [AWS Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html), with no instances or networking to manage. @@ -139,0 +81 @@ When you run your own orchestrator, you are the parent operator. You are respons + * Your instance, with AWS enforcing the boundary between workloads: use Nitro Sandboxes. This is the general case for sharing an instance between independent workloads, such as AI agents that run code from their users, workloads of different tenants, and build jobs. @@ -140,0 +83 @@ When you run your own orchestrator, you are the parent operator. You are respons + * A small, tightly scoped job on highly sensitive data: use [Nitro Enclaves](https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html). An enclave has no persistent storage, interactive access, or external networking, only a local socket to its parent instance. Choose it when that minimal interface is what your job needs. @@ -142 +85 @@ When you run your own orchestrator, you are the parent operator. You are respons -## Launch, resize, and attach storage to a Nitro Sandbox + * Your own hypervisor or operating system: use [nested virtualization](./amazon-ec2-nested-virtualization.html). @@ -144 +86,0 @@ When you run your own orchestrator, you are the parent operator. You are respons -When you run your own orchestrator, you use standard Amazon EC2 and VMM commands to launch a Nitro Sandbox, set its size, and attach storage. The examples in this section use Firecracker, the recommended VMM. For other VMMs, such as QEMU, you need to add Nitro Sandbox compatibility before you can use attestation or run a Nitro Sandbox in confidential mode. @@ -146 +87,0 @@ When you run your own orchestrator, you use standard Amazon EC2 and VMM commands -### How Firecracker works @@ -148 +88,0 @@ When you run your own orchestrator, you use standard Amazon EC2 and VMM commands -Firecracker can launch and manage Nitro Sandboxes as an alternative to KVM microVMs. Instead of creating a microVM through KVM, Firecracker uses the Nitro Enclaves driver on the parent instance to create, configure, and boot a Nitro Sandbox with a standard virtio-pci device model. Firecracker acts as the Nitro Sandbox's frontend: it owns the API socket, the device backends, and the Nitro Sandbox lifecycle. To launch a Nitro Sandbox, Firecracker does the following: @@ -150 +90 @@ Firecracker can launch and manage Nitro Sandboxes as an alternative to KVM micro - 1. Creates the Nitro Sandbox and checks the capabilities that the platform supports. +Nitro Sandboxes run on instances with Intel, AMD, and AWS Graviton processors. Nested virtualization is available only on instances with Intel processors. @@ -152,84 +92 @@ Firecracker can launch and manage Nitro Sandboxes as an alternative to KVM micro - 2. Adds vCPUs, either as dedicated cores from the parent's CPU pool, as explicit CPU IDs, or as vCPUs that share the parent's remaining cores when you enable CPU overcommit. - - 3. Allocates contiguous memory, writes the guest image into it, and donates the memory to the Nitro Sandbox. - - 4. Registers each configured virtio device, such as block, network, vsock, and entropy devices. - - 5. Starts the Nitro Sandbox. The Nitro System boots the image through UEFI firmware and presents a full PCIe topology to the guest. - - - - -After the guest starts, the device backends in Firecracker serve I/O through shared queues: guest memory in direct mode, or dedicated device memory buffers in confidential mode. The guest sees ordinary virtio-pci devices and needs no sandbox-specific code. Nitro Sandbox memory isn't overcommitted: the parent donates the full memory size when the Nitro Sandbox starts, and that memory returns to the parent only when the Nitro Sandbox terminates. - -###### Note - -Because Nitro Sandboxes build on Nitro Enclaves, the Firecracker API uses the term _enclave_ for the Nitro Sandbox configuration endpoint. Confidential mode is the default; direct mode is opt-in. - -### Launch a Nitro Sandbox - -First, launch an instance type that supports AWS Nitro Enclaves, and enable it with the `--enclave-options Enabled=true` parameter, the same option that enables Nitro Enclaves. - -