AWS eks documentation change
Summary
Added documentation for using Bottlerocket with EFA, including configuration examples and verification steps. Removed manual EFA device plugin installation steps as eksctl now handles it automatically when efaEnabled is set.
Security assessment
The changes focus on improving documentation for EFA setup with Bottlerocket and automation of device plugin deployment. There is no mention of security vulnerabilities, patches, or security-specific configurations. The updates are operational improvements rather than security-related.
Diff
diff --git a/eks/latest/userguide/node-efa.md b/eks/latest/userguide/node-efa.md index 2023214bc..39e8a7de8 100644 --- a//eks/latest/userguide/node-efa.md +++ b//eks/latest/userguide/node-efa.md @@ -110 +110 @@ If you don’t have an existing cluster, you can run the following command to cr -Because the instance type used in this example has GPUs, `eksctl` automatically installs the NVIDIA Kubernetes device plugin on each instance for you. +Because the instance type used in this example has GPUs, `eksctl` automatically installs the NVIDIA Kubernetes device plugin on each instance for you when using Amazon Linux 2. This is not necessary for Bottlerocket, as the NVIDIA device plugin is built into Bottlerocket’s EKS NVIDIA variant. When `efaEnabled` is set to `true` in the nodegroup configuration, `eksctl` will also automatically deploy the EFA device plugin on the nodes. @@ -112 +111,0 @@ Because the instance type used in this example has GPUs, `eksctl` automatically - 4. Deploy the EFA Kubernetes device plugin. @@ -114 +112,0 @@ Because the instance type used in this example has GPUs, `eksctl` automatically -The EFA Kubernetes device plugin detects and advertises EFA interfaces as allocatable resources to Kubernetes. An application can consume the extended resource type `vpc.amazonaws.com/efa` in a Pod request spec just like CPU and memory. For more information, see [Consuming extended resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#consuming-extended-resources) in the Kubernetes documentation. Once requested, the plugin automatically assigns and mounts an EFA interface to the Pod. Using the device plugin simplifies EFA setup and does not require a Pod to run in privileged mode. @@ -116,2 +114,85 @@ The EFA Kubernetes device plugin detects and advertises EFA interfaces as alloca - helm repo add eks https://aws.github.io/eks-charts - helm install aws-efa-k8s-device-plugin --namespace kube-system eks/aws-efa-k8s-device-plugin + +### Using Bottlerocket with EFA + +Bottlerocket AMI version 1.28.0 and later include official support for EFA. To use Bottlerocket for EFA-enabled nodes, specify `amiFamily: Bottlerocket` in your configuration. If you need to use a custom AMI ID, you must use standard `nodeGroups` instead of `managedNodeGroups`. + +Here’s an example configuration: + + + apiVersion: eksctl.io/v1alpha5 + kind: ClusterConfig + + metadata: + name: my-efa-bottlerocket-cluster + region: region-code + version: "1.XX" + + iam: + withOIDC: true + + availabilityZones: ["us-west-2a", "us-west-2c"] + + managedNodeGroups: + - name: my-efa-bottlerocket-ng + instanceType: p5.48xlarge + minSize: 1 + desiredCapacity: 2 + maxSize: 3 + availabilityZones: ["us-west-2a"] + volumeSize: 300 + privateNetworking: true + efaEnabled: true + amiFamily: Bottlerocket + bottlerocket: + enableAdminContainer: true + settings: + kernel: + sysctl: + "vm.nr_hugepages": "3000" # Configures 3000 * 2Mi = 6000Mi hugepages + +The `vm.nr_hugepages` sysctl setting above configures the number of 2Mi hugepages. In this example, 3000 means 3000 * 2Mi = 6000Mi of hugepages. + +### Verify EFA device plugin installation + +When you create a node group with `efaEnabled: true`, `eksctl` automatically deploys the EFA Kubernetes device plugin for you. You can verify that the device plugin is installed and functioning correctly: + + 1. Check the DaemonSet status: + + kubectl get daemonsets -n kube-system + +Sample output: + + NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE + aws-efa-k8s-device-plugin-daemonset 2 2 2 2 2 <none> 6m16s + ... + +Here, the EFA device plugin DaemonSet is running on two nodes. Both are READY and AVAILABLE. + + 2. Next, verify the pods created by the DaemonSet: + + kubectl get pods -n kube-system -l name=aws-efa-k8s-device-plugin + +Sample output: + + NAME READY STATUS RESTARTS AGE + aws-efa-k8s-device-plugin-daemonset-d68bs 1/1 Running 0 6m16s + aws-efa-k8s-device-plugin-daemonset-w4l8t 1/1 Running 0 6m16s + +The EFA device plugin pods are in a Running state, confirming that the plugin is successfully deployed and operational. + + 3. Verify resource registration: + +You can confirm that the `vpc.amazonaws.com/efa` resource is registered with the kubelet by describing the nodes: + + kubectl describe nodes + +If the EFA resource is properly registered, you will see it listed under the node’s Capacity and Allocatable resources. For example: + + Capacity: + ... + vpc.amazonaws.com/efa: 4 + Allocatable: + ... + vpc.amazonaws.com/efa: 4 + +This output confirms that the node recognizes the EFA resource, making it available for pods that request it.