AWS Security ChangesHomeSearch

AWS pcs: Rewrite AWS PCS MIG configuration guide with lifecycle-action script

Service: pcs · 2026-10-03 · Documentation medium

File: pcs/latest/userguide/mig-configuration.md · Type: isolation

Summary

Substantially restructures the MIG configuration documentation: adds a 'what a MIG node must have on every boot' section, replaces the two-step procedure with a node lifecycle action approach, provides a full idempotent bash script for partitioning GPUs, adds a 'Declare the MIG profiles to Slurm' section with a combined update example, and revises the limitations section. Notably it corrects the description of job/device isolation, stating that AWS PCS does not constrain device access through cgroups so a job can use every MIG device on the node.

Security assessment

The change is primarily operational guidance for configuring MIG on PCS compute nodes. It does document a security-relevant isolation behavior: previously the text claimed cgroups deny GPU access, but the revised text clarifies that AWS PCS does not constrain device access via cgroups, so a job allocated before a node first registers can access every MIG device and share a MIG instance with another job. This is a documented limitation rather than a fix for a specific vulnerability, so it is not flagged as security_issue_related, but it is security-adjacent guidance about GPU isolation between jobs.

Evidence

because AWS PCS doesn't constrain device access through cgroups, the job can use every MIG device on the node

Diff

diff --git a/pcs/latest/userguide/mig-configuration.md b/pcs/latest/userguide/mig-configuration.md
index 415a5df..c4ce2aa 100644
--- a//pcs/latest/userguide/mig-configuration.md
+++ b//pcs/latest/userguide/mig-configuration.md
@@ -7 +7 @@
-PrerequisitesStep 1: Configure MIG on the instancesStep 2: Configure the compute node groupLimitations
+PrerequisitesWhat a MIG node must have on every bootConfigure MIG on the instancesDeclare the MIG profiles to SlurmLimitations
@@ -14,0 +15,2 @@ MIG devices can't be described statically in `gres.conf`: their device files and
+Configuring MIG means partitioning the GPUs on each node and declaring the resulting profiles to Slurm on the compute node group. Neither works alone, so apply both in one change. If Slurm doesn't know about a node's partitions, the node has no schedulable GRES. If no node creates the declared profiles, the nodes are drained when they register. With the recommended node lifecycle action, one compute node group update carries both the partitioning script and the GRES settings.
+
@@ -26 +28,11 @@ MIG devices can't be described statically in `gres.conf`: their device files and
-## Step 1: Configure MIG on the instances
+## What a MIG node must have on every boot
+
+MIG mode and MIG instances live on the GPU, not on disk, so a custom AMI can't carry them, and they're lost whenever the instance is stopped, suspended, or terminated. On every boot, all of the following must be true before `slurmd` starts:
+
+  1. **MIG mode is enabled** on every GPU that serves MIG instances. `nvidia-smi -q` reports a `Current` and a `Pending` mode, and `Current` must be `Enabled`.
+
+  2. **The MIG instances exist** , both GPU instances and compute instances, for every profile in the compute node group's `Gres` setting.
+
+  3. **The profiles and counts match** the `Gres` setting.
+
+
@@ -28 +40,6 @@ MIG devices can't be described statically in `gres.conf`: their device files and
-The MIG instances must exist before `slurmd` starts and registers the node. If MIG mode is enabled but no MIG instances are created, `slurmd` detects the parent GPUs, finds no usable MIG device, and the node registers with no schedulable GPU. Create the MIG instances in the node's boot process, for example in a custom AMI or a launch template user data script that runs before `slurmd` starts. For more information, see [NVIDIA Multi-Instance GPU User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/).
+
+If MIG mode is enabled but no MIG instances exist, `slurmd` finds the parent GPUs but no MIG device. It logs `MIG mode is enabled, but no MIG devices were found`, then `Discarding the following config-only GPU due to lack of File specification` for each declared profile. Because the node reports fewer GPUs than `Gres` declares, the controller rejects the registration and drains the node with the reason `gres/gpu count reported lower than configured`. Jobs that request a GPU on the node fail with `Required node not available`. Creating the MIG instances afterward doesn't undrain the node; you have to resume or replace it. The node lifecycle action described in Apply the configuration with a node lifecycle action avoids this by failing the node before it registers.
+
+## Configure MIG on the instances
+
+Configuring the instances takes the following two operations. Run them from a node lifecycle action, which keeps the logic on the compute node group instead of in an AMI you have to rebuild. See Apply the configuration with a node lifecycle action. If another process already holds the GPUs at that point, run them earlier from a `cloud-init` boothook or a `systemd` unit in a custom AMI. See If MIG mode stays pending. For more information about the commands, see [NVIDIA Multi-Instance GPU User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/).
@@ -34 +51 @@ The MIG instances must exist before `slurmd` starts and registers the node. If M
-MIG mode is GPU state, not disk state, and it resets between EC2 instance allocations, so it must be enabled on every launch; it can't be baked into an AMI as already enabled. It also can't take effect while processes are using the GPUs. Enable it early in the boot sequence, before any service that uses the GPUs starts (for example, from a `systemd` unit in a custom AMI or a `cloud-init` boothook in the launch template user data; see [Example: Run early boot operations with a cloud-init boothook](./working-with_ec2-user-data_early-boot.html)). At that point no process holds the GPUs, so the mode change takes effect without a reboot. Verify that MIG mode is `Enabled` (not `Pending`) on every GPU with `nvidia-smi -q`.
+The mode change can't take effect while a process is using the GPUs. At the `nodeBootstrapped` stage `slurmd` hasn't started, so on a node that runs no other GPU service the change takes effect without a reboot. Check that the applied mode is `Enabled` on every GPU with `nvidia-smi --query-gpu=mig.mode.current --format=csv`.
@@ -38 +55 @@ MIG mode is GPU state, not disk state, and it resets between EC2 instance alloca
-If your platform does require a reboot to change MIG mode, reboot from an early boot stage such as a `cloud-init` boothook, before the [AWS PCS bootstrap](./working-with_ec2-user-data.html) starts. Don't reboot from a node lifecycle action script: a reboot there interrupts the AWS PCS bootstrap sequence. For how to create the boot hook, see [Example: Run early boot operations with a cloud-init boothook](./working-with_ec2-user-data_early-boot.html).
+If MIG mode stays `Pending`, another process has the GPUs open. See If MIG mode stays pending.
@@ -43 +60 @@ If your platform does require a reboot to change MIG mode, reboot from an early
-        nvidia-smi mig -i $i -cgi 9,14,19,19 -C
+        nvidia-smi mig -i $i -cgi 3g.20gb,2g.10gb,1g.5gb,1g.5gb -C
@@ -45,0 +63 @@ If your platform does require a reboot to change MIG mode, reboot from an early
+`-cgi` also accepts numeric profile IDs, but the same ID means a different profile on a different GPU model: `9,14,19,19` is this layout on the 40 GB A100 in a `p4d` instance, and `3g.40gb`, `2g.20gb`, and two `1g.10gb` on an 80 GB part. Use the profile names, so that they match the `Gres` setting and creation fails on a GPU that has no such profile.
@@ -46,0 +65 @@ If your platform does require a reboot to change MIG mode, reboot from an early
+Create the largest profiles first, as in the example. MIG placements are fixed, so a layout that fits can still fail to create if the small profiles take the slices the large ones need.
@@ -49 +67,0 @@ If your platform does require a reboot to change MIG mode, reboot from an early
-## Step 2: Configure the compute node group
@@ -51 +68,0 @@ If your platform does require a reboot to change MIG mode, reboot from an early
-Declare both halves of the GRES configuration on the compute node group:
@@ -53 +70 @@ Declare both halves of the GRES configuration on the compute node group:
-  * In `gresCustomSettings`, a GPU record that only enables `AutoDetect`. Don't set `Name`, `Type`, or `File`. The MIG devices are discovered on the node.
+### Apply the configuration with a node lifecycle action
@@ -55 +72 @@ Declare both halves of the GRES configuration on the compute node group:
-  * In `slurmCustomSettings`, a `Gres` setting that declares the MIG profiles and counts that jobs can request. The profile names and counts must match the MIG instances created on the node.
+Configure the node lifecycle action as follows. For how to add one to a compute node group, see [Configure node lifecycle actions in AWS PCS](./cng-node-lifecycle-actions-configure.html).
@@ -56,0 +74,6 @@ Declare both halves of the GRES configuration on the compute node group:
+**Field** | **Value** | **Why**  
+---|---|---  
+Lifecycle stage | `nodeBootstrapped` | Runs before `slurmd` opens the GPUs and registers the node.  
+`executionPolicy` | `EVERY_BOOT` | MIG state doesn't survive a stop, suspend, or termination.  
+`arguments` | `["3g.20gb,2g.10gb,1g.5gb,1g.5gb"]` | The MIG profiles to create on each GPU, by name. One script then serves compute node groups with different layouts.  
+`onError` | `TERMINATE` | Replaces a node that can't be partitioned instead of letting it register without its MIG devices. While debugging, use `STOP_SEQUENCE` to keep the instance for inspection.  
@@ -57,0 +81 @@ Declare both halves of the GRES configuration on the compute node group:
+A script that runs on every boot must be idempotent. The following outline takes the per-GPU layout as its first argument and handles one GPU at a time. It skips a GPU that already has that layout, repartitions any other, and exits non-zero if it can't finish, which fails the node. It never reboots the node. For how arguments reach a script, see [Passing arguments to scripts](./cng-node-lifecycle-actions-configure.html#cng-node-lifecycle-actions-configure-arguments).
@@ -60 +84,4 @@ Declare both halves of the GRES configuration on the compute node group:
-###### Example– Declaring MIG profiles on a compute node group (8 GPUs partitioned as shown in Step 1)
+    #!/usr/bin/env bash
+    # Partition every GPU as $1 specifies, then verify the result. Expected
+    # nvidia-smi failures are handled inline, so this doesn't use 'set -e'.
+    set -uo pipefail
@@ -61,0 +89,8 @@ Declare both halves of the GRES configuration on the compute node group:
+    # $1 - per-GPU MIG profile names, comma-separated, largest first, as in Gres.
+    PROFILES="${1:-}"
+    if ! echo "$PROFILES" | grep -Eq '^[0-9]+g\.[0-9]+gb(,[0-9]+g\.[0-9]+gb)*$'; then
+        echo "ERROR: pass the per-GPU MIG profiles as the first argument," >&2
+        echo "       for example 3g.20gb,2g.10gb,1g.5gb,1g.5gb" >&2
+        exit 1
+    fi
+    EXPECTED=$(echo "$PROFILES" | tr ',' '\n' | sort)
@@ -63,5 +98,9 @@ Declare both halves of the GRES configuration on the compute node group:
-    aws pcs update-compute-node-group \
-        --cluster-identifier my-cluster \
-        --compute-node-group-identifier my-cng-1 \
-        --slurm-configuration \
-        'gresCustomSettings=[{AutoDetect=nvml}],slurmCustomSettings=[{parameterName=Gres,parameterValue=gpu:3g.20gb:8\,gpu:2g.10gb:8\,gpu:1g.5gb:16}]'
+    # Fail, rather than exit 0, when there is no GPU to configure.
+    GPUS=$(nvidia-smi --query-gpu=index --format=csv,noheader) || {
+        echo "ERROR: nvidia-smi failed. This node has no usable NVIDIA driver." >&2
+        exit 1
+    }
+    if [ -z "$GPUS" ]; then
+        echo "ERROR: no NVIDIA GPUs detected." >&2
+        exit 1
+    fi
@@ -69 +108,15 @@ Declare both halves of the GRES configuration on the compute node group:
-For the constraints between the two halves, see [Constraints between gres.conf and the Gres setting](./gres-custom-settings.html#gres-custom-settings-constraints). Because AWS PCS can't count devices that are discovered at boot, it accepts any profile names and counts in the `Gres` setting for a record that only enables `AutoDetect`. Consistency with the actual MIG instances is only verified when the node registers.
+    # The MIG profiles on GPU $1, in the same form as EXPECTED. nvidia-smi -L lists
+    # what NVML reports, which is what slurmd autodetects.
+    mig_devices() {
+        nvidia-smi -L | awk -v gpu="$1" '
+            /^GPU [0-9]+:/ { current = ($2 + 0); next }
+            $1 == "MIG" && current == gpu { print $2 }
+        ' | sort
+    }
+    
+    for i in $GPUS; do
+        # 1. Skip a GPU that already has the requested layout.
+        if [ "$(mig_devices "$i")" = "$EXPECTED" ]; then
+            echo "GPU $i: already configured as $PROFILES"
+            continue
+        fi
@@ -71 +124,18 @@ For the constraints between the two halves, see [Constraints between gres.conf a
-When a node boots and registers, it advertises the discovered profiles with their socket affinity, for example `Gres=gpu:3g.20gb:8(S:0),gpu:1g.5gb:28(S:1)`. A job that requests a profile, such as `--gres=gpu:3g.20gb:1`, is placed on a single MIG instance and gets its UUID in `CUDA_VISIBLE_DEVICES` (for example, `MIG-e7aa6185-06d5-53e9-a4d9-00f7670f741e`), so CUDA restricts the job to that partition.
+        # 2. Enable MIG mode. Ignore the exit status; step 3 checks the result.
+        enable_output=$(nvidia-smi -i "$i" -mig 1 2>&1) || true
+    
+        # 3. Fail if the mode isn't applied. Don't reboot: that would interrupt
+        #    the AWS PCS bootstrap.
+        current=$(nvidia-smi -i "$i" --query-gpu=mig.mode.current --format=csv,noheader)
+        pending=$(nvidia-smi -i "$i" --query-gpu=mig.mode.pending --format=csv,noheader)
+        if [ "$current" != "Enabled" ]; then
+            echo "ERROR: GPU $i MIG mode is '$current', pending '$pending'." >&2
+            echo "$enable_output" >&2
+            if [ "$pending" = "Enabled" ]; then
+                echo "The enable is pending, so a process is holding the GPU. Stop it," >&2
+                echo "or configure MIG mode in a boothook instead." >&2
+            elif [ "$current" = "[N/A]" ] || [ "$current" = "N/A" ]; then
+                echo "This GPU does not support MIG." >&2
+            fi
+            exit 1
+        fi
@@ -73 +143,8 @@ When a node boots and registers, it advertises the discovered profiles with thei
-###### Note
+        # 4. Recreate the layout from scratch, because instances can't be added to
+        #    a partial one. Nothing uses them yet, because slurmd hasn't started.
+        nvidia-smi mig -i "$i" -dci >/dev/null 2>&1 || true
+        nvidia-smi mig -i "$i" -dgi >/dev/null 2>&1 || true
+        if ! nvidia-smi mig -i "$i" -cgi "$PROFILES" -C; then
+            echo "ERROR: MIG instance creation failed on GPU $i" >&2
+            exit 1
+        fi
@@ -75 +152,11 @@ When a node boots and registers, it advertises the discovered profiles with thei
-The controller logs warnings such as `Ignoring file-less GPU gpu:3g.20gb from final GRES list` for MIG records. This is expected: the authoritative GRES comes from node registration.
+        # 5. Check which profiles exist, not only how many, so a layout that
+        #    doesn't match Gres fails here instead of at job time.
+        actual=$(mig_devices "$i")
+        if [ "$actual" != "$EXPECTED" ]; then
+            echo "ERROR: GPU $i has the wrong MIG layout." >&2
+            echo "  expected: $(echo "$EXPECTED" | tr '\n' ' ')" >&2
+            echo "  found:    $(echo "$actual" | tr '\n' ' ')" >&2
+            exit 1
+        fi
+        echo "GPU $i: configured as $PROFILES"
+    done
@@ -77 +164 @@ The controller logs warnings such as `Ignoring file-less GPU gpu:3g.20gb from fi
-## Limitations
+To set the script and the GRES settings in one call, see the example in Declare the MIG profiles to Slurm.
@@ -79 +166 @@ The controller logs warnings such as `Ignoring file-less GPU gpu:3g.20gb from fi
-Because a MIG configuration names no devices, the controller only knows a node's real GRES after the node boots and registers. This causes the following limitations. They apply to any GPU record that only enables `AutoDetect`, and therefore to every MIG compute node group.
+If you adapt the script, keep both checks. Without the mode check, a node whose MIG mode never became `Enabled` goes on to register without its MIG devices, and the cause shows only in the `slurmd` log. Without the layout check, a node that creates the wrong profiles registers normally and fails only the jobs that request the declared profiles. For where the script's output is written, see [Logging and debugging](./cng-node-lifecycle-actions-configure.html#cng-node-lifecycle-actions-configure-logging).
@@ -81 +168,12 @@ Because a MIG configuration names no devices, the controller only knows a node's
-###### Note
+### If MIG mode stays pending
+
+Enabling MIG mode fails while any process has the GPUs open. `nvidia-smi` reports the enable as pending, for example with `In use by another client`, and it takes effect only after those processes exit or the node reboots. On a AWS PCS compute node the process is usually a GPU monitoring agent, such as NVIDIA DCGM or an agent that ships in the AMI, or `slurmd` itself if MIG is configured after `slurmd` starts. You can't reboot from the lifecycle action, so use one of the following fixes:
+
+  * **Stop the process that holds the GPUs.** In the custom AMI, disable or remove any daemon that opens the GPUs before MIG configuration runs. Also remove any daemon that configures MIG itself. Two components partitioning the same GPUs produce a layout that doesn't match the `Gres` setting.
+
+  * **Configure MIG in a boothook.** If you can't stop the process, move the MIG mode change and the profile creation into a `cloud-init` boothook in the launch template user data. A boothook runs before the AWS PCS bootstrap and before the AMI's services start, so the mode change takes effect without a reboot. For how to write one, see [Example: Run early boot operations with a cloud-init boothook](./working-with_ec2-user-data_early-boot.html). Keep the lifecycle action as a check: on every boot it confirms the layout and fails the node if the boothook didn't apply it.
+
+
+
+
+## Declare the MIG profiles to Slurm
@@ -83 +181 @@ Because a MIG configuration names no devices, the controller only knows a node's
-As a best practice, use a static compute node group for MIG: set `minInstanceCount` equal to `maxInstanceCount` in the scaling configuration. The nodes then stay running and registered instead of powering down between jobs, which mitigates both limitations.
+Declare both GRES settings on the compute node group:
@@ -85 +183 @@ As a best practice, use a static compute node group for MIG: set `minInstanceCou
-### Configuration updates don't reach powered-down nodes
+  * In `gresCustomSettings`, a GPU record that only enables `AutoDetect`. Don't set `Name`, `Type`, or `File`. The MIG devices are discovered on the node.
@@ -87 +185 @@ As a best practice, use a static compute node group for MIG: set `minInstanceCou
-When you change the MIG configuration of a compute node group (the profiles in the `Gres` setting, or the MIG partitioning in the boot process), the change doesn't reach nodes that previously registered the old configuration and are powered down at that moment. Those nodes keep advertising the profiles they last registered until they next boot and register, which only happens when a job is allocated to them. The consequences are:
+  * In `slurmCustomSettings`, a `Gres` setting that declares the MIG profiles and counts that jobs can request. The profile names and counts must match the MIG instances created on the node.
@@ -89 +186,0 @@ When you change the MIG configuration of a compute node group (the profiles in t
-  * Jobs that request the new profiles are rejected at submission with `Requested node configuration is not available`, because no node advertises them yet.
@@ -91 +187,0 @@ When you change the MIG configuration of a compute node group (the profiles in t
-  * Jobs that request the old profiles are still accepted, even though the configuration no longer declares them.
@@ -93 +188,0 @@ When you change the MIG configuration of a compute node group (the profiles in t
-  * If you reduce a profile count, a job sized for the old count can be admitted and run on a node that has fewer MIG instances than the job requested.
@@ -94,0 +190 @@ When you change the MIG configuration of a compute node group (the profiles in t
+###### Example– Applying both parts in one update (8 GPUs, each partitioned into one `3g.20gb`, one `2g.10gb`, and two `1g.5gb`)
@@ -96,0 +193,34 @@ When you change the MIG configuration of a compute node group (the profiles in t
+    aws pcs update-compute-node-group \
+        --cluster-identifier my-cluster \
+        --compute-node-group-identifier my-cng-1 \
+        --slurm-configuration '{
+          "gresCustomSettings": [
+            { "AutoDetect": "nvml" }
+          ],
+          "slurmCustomSettings": [
+            { "parameterName": "Gres", "parameterValue": "gpu:3g.20gb:8,gpu:2g.10gb:8,gpu:1g.5gb:16" }