AWS eks: Add EKS NodeClass kubelet and kernel sysctl configuration docs
Summary
Documents new advancedCompute.kubelet and advancedCompute.kernel.sysctl fields, transparent hugepages, and restrictions for unsafe sysctls and EFA/hugepages mutual exclusion.
Security assessment
The diff adds documentation for kubelet and kernel sysctl settings, including restrictions on allowedUnsafeSysctls and user namespace requirements. This is security-adjacent hardening guidance but does not reference a specific vulnerability, CVE, or incident.
Evidence
+ * **`allowedUnsafeSysctls` restrictions**: You can specify up to 64 entries, and each entry must match one of the allowed patterns: `kernel.shm*`, `kernel.msg*`, `kernel.sem`, `fs.mqueue.****`, or `net.`. Allowing an unsafe sysctl on the node only permits Pods to request it. Each Pod must still list the sysctl in its `securityContext.sysctls`. For more information, see [Using sysctls in a Kubernetes Cluster](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/) in the Kubernetes documentation.
Diff
diff --git a/eks/latest/userguide/create-node-class.md b/eks/latest/userguide/create-node-class.md index f04bd5219..6a15b8110 100644 --- a//eks/latest/userguide/create-node-class.md +++ b//eks/latest/userguide/create-node-class.md @@ -7 +7 @@ -Considerations for pod-sparse or high-scale workloadsCreate a Node ClassCreate node class access entryNode Class SpecificationConsiderationsSeparate subnets and security groups for PodsSecondary IP Mode for PodsDisable IPv4 egress from IPv6 pods in IPv6 clusters.Static Network Interface ConfigurationConfigure hugepages on nodes +Considerations for pod-sparse or high-scale workloadsCreate a Node ClassCreate node class access entryNode Class SpecificationConsiderationsSeparate subnets and security groups for PodsSecondary IP Mode for PodsDisable IPv4 egress from IPv6 pods in IPv6 clusters.Static Network Interface ConfigurationConfigure hugepages on nodesConfigure kubelet settings on nodesConfigure kernel sysctls on nodes @@ -297,0 +298,4 @@ For information about deploying CloudFormation stacks, see [Getting started with + * **Kubelet settings** — For information about overriding kubelet settings such as Pod density, eviction thresholds, and log rotation, see Configure kubelet settings on nodes. + + * **Kernel sysctls** – For information about configuring kernel sysctls on nodes, see Configure kernel sysctls on nodes. + @@ -540 +544,8 @@ The following example shows a `NodeClass` configured with EFA-only interfaces fo -Use the `advancedCompute.hugepages` field to pre-allocate hugepages on nodes in the `NodeClass`. Hugepages improve memory-access performance for latency-sensitive workloads such as high-performance computing (HPC), databases, and network-intensive applications. To consume hugepages, configure your Pods to explicitly request hugepage resources. For more information, see [Manage HugePages](https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/) in the Kubernetes documentation. +With the `advancedCompute.hugepages` field, you can configure hugepages on nodes in the `NodeClass`. Hugepages improve memory-access performance for latency-sensitive workloads such as high-performance computing (HPC), databases, and network-intensive applications. You can use two independent hugepages configurations separately or together: + + * **Static hugepages** – Use the `pages` field to pre-allocate a fixed number of `2Mi` or `1Gi` hugepages on each node, or a mix of both sizes. Static hugepages are a schedulable node resource. To consume them, configure your Pods to explicitly request hugepage resources such as `hugepages-2Mi` or `hugepages-1Gi`. For more information, see [Manage HugePages](https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/) in the Kubernetes documentation. + + * **Transparent hugepages (THP)** – Use the `transparent` field to configure the kernel’s THP policy. THP is a kernel-wide policy rather than an allocatable resource. It does not affect scheduling or node capacity, and Pods do not request it. For more information, see [Transparent Hugepage Support](https://docs.kernel.org/admin-guide/mm/transhuge.html) in the Linux kernel documentation. + + + @@ -561,0 +573 @@ Use the `advancedCompute.hugepages` field to pre-allocate hugepages on nodes in + # Static hugepage reservations: up to one entry for each size @@ -564,0 +577,19 @@ Use the `advancedCompute.hugepages` field to pre-allocate hugepages on nodes in + - size: "1Gi" + count: 2 + # Transparent hugepages (THP) kernel policy + transparent: + enabled: always # or madvise, never + defrag: defer # or always, defer+madvise, madvise, never + +### Transparent hugepage settings + +Both `transparent` settings are optional. If you omit a setting, the node uses the kernel default (`madvise`). + + * **`enabled` ** – Use `enabled` to control the THP allocation policy. Set this to `always` to apply THP system-wide. Set it to `madvise` to apply THP only to memory regions that opt in with `madvise(MADV_HUGEPAGE)`. Set it to `never` to disable THP. + + * **`defrag` ** – Use `defrag` to control how aggressively the kernel compacts memory to satisfy a THP allocation. Valid values are `always`, `defer`, `defer+madvise`, `madvise`, and `never`. + + + + +To disable THP, such as for database workloads that recommend turning it off, set `enabled: never`. When `enabled` is `never`, leave `defrag` unset or set it to `never`; NodeClass validation rejects any other combination. @@ -570 +601,180 @@ Consider the following when configuring hugepages on a NodeClass: - * **Memory reservation** : Hugepages count against the instance’s total memory. Amazon EKS marks instance types as incompatible with the NodeClass when the hugepages reservation would exceed 80% of total memory. + * **Static hugepage sizes** – Use the `pages` field to specify up to one entry for each hugepage size (`2Mi` and `1Gi`). You can reserve both sizes on the same nodes. + + * **Memory reservation** – Static hugepages count against the instance’s total memory. Amazon EKS marks instance types as incompatible with the NodeClass when the combined hugepages reservation across all page sizes would exceed 80% of total memory. + + * **EFA mutual exclusion** – Elastic Fabric Adapter (EFA) and static hugepages cannot coexist on the same NodeClass. If you set `interfaceType: efa-only` together with `pages`, NodeClass validation fails. Amazon EKS does not provision Pods that request `vpc.amazonaws.com/efa` from a NodeClass with static hugepages. Use a separate NodeClass for EFA workloads. Transparent hugepages do not conflict with EFA; you can configure `transparent` on a NodeClass that uses EFA interfaces. + + + + +## Configure kubelet settings on nodes + +Use the `advancedCompute.kubelet` field to override kubelet settings on nodes in the `NodeClass`. With these overrides, you can tune Pod density, resource-pressure eviction, container log rotation, and out-of-memory (OOM) behavior. Amazon EKS applies default kubelet settings to Auto Mode nodes, so configure this field only when your workloads require different behavior. For more information about the underlying settings, see [Kubelet Configuration](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/) in the Kubernetes documentation. + +The following kubelet settings are available: + + * `maxPods`: The maximum number of Pods that can run on a node. + + * `podPidsLimit`: The maximum number of process IDs (PIDs) per Pod. + + * `idsPerPod`: The number of UIDs and GIDs the kubelet allocates to each Pod’s user namespace. + + * `singleProcessOOMKill`: Whether the kubelet OOM kills processes in a container individually instead of as a group. + + * `eviction`: Hard and soft eviction thresholds for resource-pressure signals. + + * `logging`: Container log rotation size and file count. + + * `allowedUnsafeSysctls`: A list of unsafe sysctls, or namespaced patterns ending in `*`, that Pods scheduled on the node are permitted to set. + + + + +### Example configuration + + + apiVersion: eks.amazonaws.com/v1 + kind: NodeClass + metadata: + name: kubelet-compute + spec: + role: MyNodeRole + + subnetSelectorTerms: + - tags: + Name: "private-subnet" + + securityGroupSelectorTerms: + - tags: + Name: "eks-cluster-sg" + + advancedCompute: + kubelet: + # Range: 1-110 + maxPods: 60 + # -1 disables the limit, or specify 100 or greater + podPidsLimit: 4096 + # Multiple of 65536, range: 65536-4294967295. Requires + # advancedCompute.kernel.sysctl.user.max_user_namespaces > 0. + idsPerPod: 65536 + singleProcessOOMKill: false + eviction: + # Valid signals: memory.available, nodefs.available, nodefs.inodesFree, + # imagefs.available, imagefs.inodesFree, pid.available + hard: + memory.available: "100Mi" + nodefs.available: "10%" + soft: + memory.available: "500Mi" + nodefs.available: "15%" + # Required when soft thresholds are set + softGracePeriod: + memory.available: "1m30s" + nodefs.available: "2m" + # 0 uses each Pod's terminationGracePeriodSeconds + maxPodGracePeriod: 60 + logging: + containerLogMaxSize: "10Mi" + # Minimum: 2 + containerLogMaxFiles: 5 + # Up to 64 entries. Allowed patterns: kernel.shm*, kernel.msg*, + # kernel.sem, fs.mqueue.*, net.* + allowedUnsafeSysctls: + - "net.core.somaxconn" + - "kernel.shm*" + +### Kubelet settings considerations + +Consider the following when configuring kubelet settings on a NodeClass: + + * **Node replacement** : Changing kubelet settings on an existing `NodeClass` marks the nodes that use it as drifted. Amazon EKS replaces those nodes with new nodes that use the updated configuration. Amazon EKS does not apply kubelet settings to running nodes in place. + + * **`maxPods` clamping**: The value you set is an upper bound, not a guarantee. At runtime, Amazon EKS applies the lowest of three values. These are your `maxPods` value, the IP addresses available for the instance type, and the Auto Mode limit of 110 Pods per node. For more information, see [Choose an optimal Amazon EC2 node instance type](./choosing-instance-type.html). + + * **`podPidsLimit` values**: Specify `-1` to disable the limit, or a value of `100` or greater. `NodeClass` validation rejects values between `0` and `99`. + + * **`idsPerPod` requirements**: The value must be a multiple of `65536` between `65536` and `4294967295`. Setting `idsPerPod` requires `advancedCompute.kernel.sysctl.user.max_user_namespaces` to be set to a positive value on the same `NodeClass`; `NodeClass` admission rejects configurations that set `idsPerPod` without also setting `user.max_user_namespaces` to a value greater than `0`. For more information about `user.max_user_namespaces`, see Supported sysctls. + + * **Soft eviction thresholds** : A soft threshold must be less aggressive than the hard threshold for the same signal, meaning it triggers earlier. If you set soft thresholds, you must also set `softGracePeriod` for each signal. A soft threshold and hard threshold can use different units, such as a percentage and an absolute quantity. In that case, Amazon EKS resolves both values for each instance type. Amazon EKS then marks instance types that violate the comparison as incompatible with the `NodeClass`. + + * **`singleProcessOOMKill` version requirement**: This setting requires Kubernetes version 1.32 or later. On earlier versions, Amazon EKS emits a warning event on the `NodeClass` and the setting has no effect. + + * **`allowedUnsafeSysctls` restrictions**: You can specify up to 64 entries, and each entry must match one of the allowed patterns: `kernel.shm*`, `kernel.msg*`, `kernel.sem`, `fs.mqueue.****`, or `net.`. Allowing an unsafe sysctl on the node only permits Pods to request it. Each Pod must still list the sysctl in its `securityContext.sysctls`. For more information, see [Using sysctls in a Kubernetes Cluster](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/) in the Kubernetes documentation. + + + + +## Configure kernel sysctls on nodes + +Use the `advancedCompute.kernel.sysctl` field to set Linux kernel sysctls on nodes in the `NodeClass`. Sysctls tune runtime kernel behavior for workloads that need higher networking limits, different memory-management policies, or specific TCP settings. Property names on the `sysctl` field exactly match the underlying kernel sysctl name (for example, `vm.swappiness` and `net.core.somaxconn`). Amazon EKS applies sysctl values at node boot. If you change a sysctl on an existing NodeClass, Amazon EKS begins replacing nodes for the new values to take effect. + +### Example configuration + + + apiVersion: eks.amazonaws.com/v1 + kind: NodeClass + metadata: + name: tuned-compute + spec: + role: MyNodeRole + + subnetSelectorTerms: + - tags: + Name: "private-subnet" + + securityGroupSelectorTerms: + - tags: + Name: "eks-cluster-sg" + + advancedCompute: + kernel: + sysctl: + vm.swappiness: 30 + net.core.somaxconn: 4096 + net.ipv4.tcp_congestion_control: "cubic" + vm.dirty_background_ratio: 10 + vm.dirty_ratio: 20 + +### Supported sysctls + +Amazon EKS supports only the sysctls in the following table. Each numeric sysctl enforces the minimum and maximum shown, and NodeClass admission rejects values outside that range. String sysctls accept only the listed values. + +Sysctl | Description | Allowed values +---|---|--- +`fs.aio-max-nr` | The maximum number of asynchronous I/O requests. | `65536`–`4194304` +`fs.inotify.max_user_instances` | The maximum number of inotify instances per real user ID. | `8192`–`1048576` +`fs.inotify.max_user_watches` | The maximum number of inotify watches per real user ID. | `8192`–`1048576` +`fs.nr_open` | The maximum number of file handles a process can allocate. | `1048576`–`2147483584` +`kernel.perf_event_paranoid` | Controls unprivileged access to performance event systems. | `-1`–`2` +`net.core.netdev_max_backlog` | The maximum number of packets queued on the INPUT side per CPU. | `1000`–`3240000` +`net.core.rmem_max` | The maximum receive socket buffer size, in bytes. | `212992`–`134217728` +`net.core.somaxconn` | The maximum value for the backlog argument in the listen syscall. | `4096`–`3240000` +`net.core.wmem_max` | The maximum send socket buffer size, in bytes. | `212992`–`134217728` +`net.ipv4.neigh.default.gc_thresh1` | Minimum ARP-cache entries before garbage collection can run. | `0`–`262144` +`net.ipv4.neigh.default.gc_thresh2` | Soft maximum number of ARP-cache entries. | `512`–`524288`