AWS sagemaker: Enhanced EKS cluster access setup with IAM policies and namespace restrictions
Summary
Updated documentation for configuring EKS cluster access in SageMaker HyperPod Studio, emphasizing IAM policy requirements, cluster-access policies, and namespace-based task visibility restrictions.
Security assessment
The evidence line explicitly documents the need for cluster-access policies to enforce Kubernetes permissions, addressing authorization risks by preventing excessive privileges within the cluster.
Evidence
+ 3. Attach cluster-access policies to the execution role. The IAM policy in the previous step lets the execution role call the AWS APIs. It does not give the role any permissions inside the Kubernetes cluster. Cluster-access policies do that, and only a role with the right ones attached reaches the cluster. Attach them from **Manage access** , on the domain or the user profile.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-studio-setup-eks.md b/sagemaker/latest/dg/sagemaker-hyperpod-studio-setup-eks.md index 30b09202f..fafdbbd71 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-studio-setup-eks.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-studio-setup-eks.md @@ -10,0 +11,6 @@ Restrict task view in Studio for EKS clusters +You do most of this setup from your HyperPod cluster details page in the SageMaker AI console. Open the SageMaker AI console, choose **HyperPod clusters** , choose your cluster, and then choose the **Configuration** tab. Under **Cluster access for SageMaker domains** , choose **Manage access**. This is where you create or view a domain and attach the cluster-access policies that let Studio users reach the cluster. + +The following screenshot shows the **Cluster access for SageMaker domains** section on the **Configuration** tab. + + + @@ -13 +19 @@ The following instructions describe how to set up an Amazon EKS cluster in Studi - 1. Create a domain or have one ready. For information on creating a domain, see [Guide to getting set up with Amazon SageMaker AI](./gs.html). + 1. Under the **Manage access** page, select an existing domain. Studio access to a HyperPod cluster runs through a domain, and the domain execution role is the IAM principal Studio uses to act on your cluster. For information on creating a domain, see [Guide to getting set up with Amazon SageMaker AI](./gs.html). @@ -15 +21 @@ The following instructions describe how to set up an Amazon EKS cluster in Studi - 2. Add the following permission to your execution role. + 2. Attach the following permissions to your execution role from the IAM console. @@ -21 +27 @@ To learn how to attach policies to an IAM user or group, see [Adding and removin -JSON +Before you attach the policy, replace both example ARNs with your own: @@ -22,0 +29 @@ JSON + * Replace `arn:aws:sagemaker:us-east-1:111122223333:cluster/hyperpod-cluster-name` with your HyperPod cluster ARN. @@ -24 +31 @@ JSON -**** + * Replace `arn:aws:eks:us-east-1:111122223333:cluster/eks-cluster-name` with your Amazon EKS cluster ARN. It appears twice, in `UseEksClusterPermissions` and in `DescribeSpacesAddon`, where it carries a trailing `/*`. @@ -25,0 +33 @@ JSON +These are two different resources with two different ARNs. Find the HyperPod cluster ARN in the SageMaker AI console and the Amazon EKS cluster ARN in the Amazon EKS console. If you leave the example values in place, Studio cannot describe your cluster and the **Tasks** tab does not load. @@ -31 +39 @@ JSON - "Sid": "DescribeHyerpodClusterPermissions", + "Sid": "DescribeHyperpodClusterPermissions", @@ -36 +44 @@ JSON - "Resource": "arn:aws:sagemaker:us-east-1:111122223333:cluster/cluster-name" + "Resource": "arn:aws:sagemaker:us-east-1:111122223333:cluster/hyperpod-cluster-name" @@ -69 +77 @@ JSON - "eks:DescribeAddon" + "eks:MutateViaKubernetesApi" @@ -71 +79,7 @@ JSON - "Resource": "arn:aws:eks:us-east-1:111122223333:cluster/cluster-name" + "Resource": "arn:aws:eks:us-east-1:111122223333:cluster/eks-cluster-name" + }, + { + "Sid": "DescribeSpacesAddon", + "Effect": "Allow", + "Action": "eks:DescribeAddon", + "Resource": "arn:aws:eks:us-east-1:111122223333:cluster/eks-cluster-name/*" @@ -91,0 +106,3 @@ JSON + 3. Attach cluster-access policies to the execution role. The IAM policy in the previous step lets the execution role call the AWS APIs. It does not give the role any permissions inside the Kubernetes cluster. Cluster-access policies do that, and only a role with the right ones attached reaches the cluster. Attach them from **Manage access** , on the domain or the user profile. + +Select the execution role for the domain or user profile you are granting access to, then select the policies your users need. Choose whether to scope access to a namespace or to the whole cluster, scoping to a namespace when teams share a cluster, and then save. Scoping to a namespace also restricts which tasks those users can see in Studio. For what each policy allows, how to scope access, and how to grant a custom set of permissions instead, see Restrict task view in Studio for EKS clusters. @@ -93 +110 @@ JSON - 3. [Grant IAM users access to Kubernetes with EKS access entries](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html). +Granting access this way creates the Amazon EKS access entry for the execution role for you, so there is no separate step in the Amazon EKS console. For the concepts behind the access model, see [Grant IAM users access to Kubernetes with EKS access entries](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html). @@ -95 +112 @@ JSON - 1. Navigate to the Amazon EKS cluster associated with your HyperPod cluster. + 4. (Optional) To ensure a more smooth experience, we recommend that you add tags to your clusters. For information on how to add tags, see [Edit a SageMaker HyperPod cluster](./sagemaker-hyperpod-operate-slurm-console-ui.html#sagemaker-hyperpod-operate-slurm-console-ui-edit-clusters) to update your cluster using the SageMaker AI console. @@ -97 +114 @@ JSON - 2. Choose the **Access** tab and [create an access entry](https://docs.aws.amazon.com/eks/latest/userguide/creating-access-entries.html) for the execution role you created. +Tag your [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/what-is-Amazon-Managed-Service-Grafana.html) workspace to your Studio domain. Use this tag to link to your Grafana workspace directly from your cluster in Studio. Add the following tag to your cluster to identify it with your Grafana workspace ID, `ws-id`. @@ -99 +116 @@ JSON - 1. In step 1, Select the execution role you created above in the **IAM** principal dropdown. +Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. @@ -101 +117,0 @@ JSON - 2. In step 2, select a policy name and select an access scope that you want the users to have access to. @@ -103 +118,0 @@ JSON - 4. (Optional) To ensure a more smooth experience, we recommend that you add tags to your clusters. For information on how to add tags, see [Edit a SageMaker HyperPod cluster](./sagemaker-hyperpod-operate-slurm-console-ui.html#sagemaker-hyperpod-operate-slurm-console-ui-edit-clusters) to update your cluster using the SageMaker AI console. @@ -105 +119,0 @@ JSON - 1. Tag your [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/what-is-Amazon-Managed-Service-Grafana.html) workspace to your Studio domain. This will be used to quickly link to your Grafana workspace directly from your cluster in Studio. To do so, add the following tag to your cluster to identify it with your Grafana workspace ID, `ws-id`. @@ -107 +121 @@ JSON -Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. +## Restrict task view in Studio for EKS clusters @@ -109 +123 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. - 5. (Optional) Restrict task view in Studio for EKS clusters. For information on viewable tasks in Studio, see [Tasks](./sagemaker-hyperpod-studio-tabs.html#sagemaker-hyperpod-studio-tabs-tasks). +You can restrict users’ visibility to specified Kubernetes namespaces, ensuring that users can access the resources they need while maintaining strict access controls. @@ -110,0 +125 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. +There are two ways to do this. Scoping a cluster-access policy to a namespace is done entirely in the console and is the simpler option. A custom Kubernetes RBAC role gives you control over the exact verbs and resources a user gets, at the cost of managing the role yourself. @@ -111,0 +127 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. +### Restrict with a cluster-access policy @@ -112,0 +129 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. +HyperPod provides cluster-access policies that you attach from **Manage access** on the **Configuration** tab. Attach only the policies a set of users needs, and scope the access to a namespace rather than to the whole cluster. This is the same flow as the third step above. @@ -114 +131,9 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. -## Restrict task view in Studio for EKS clusters +We recommend that you attach all of the policies to the role for the full HyperPod in Studio experience. Each policy covers a different part of the experience, so leaving one off removes the capability it grants. Attach a subset only when you intend to withhold a capability from that set of users. + +Policy | What it allows +---|--- +`AmazonSagemakerHyperpodTrainingPolicy` | Submit and manage training workloads. Full access to `RayCluster`, `RayJob`, and `RayCronJob`, to `HyperPodPyTorchJob`, to Kubeflow `PyTorchJob`, `MPIJob`, and `TFJob`, and to Kubernetes jobs and pods. Read access to pod logs, config maps, events, services, service accounts, resource quotas, limit ranges, deployments, stateful sets, replica sets, and Kueue local queues and workloads. Can create a `RayDashboardConnection`. +`AmazonSagemakerHyperpodInferencePolicy` | Deploy and manage inference workloads. Full access to `RayCluster` and `RayService`, to `JumpStartModel`, `InferenceEndpointConfig`, and `SageMakerEndpointRegistration`, and to pods. Read access to pod logs, config maps, events, services, service accounts, resource quotas, limit ranges, deployments, stateful sets, replica sets, horizontal pod autoscalers, ingresses, and Kueue local queues and workloads. Can create a `RayDashboardConnection`. +`AmazonSagemakerHyperpodSpacePolicy` | Use spaces for interactive development. Full access to `Workspace` resources, and read access to workspace templates, access strategies, and integration templates. Read access to pods, services, service accounts, persistent volume claims, events, resource quotas, bindings, daemon sets, deployments, and replica sets. Can create a `WorkspaceConnection`, which is what opens a space. +`AmazonSagemakerHyperpodSpaceTemplatePolicy` | Read the shared space templates. Attach it scoped to the `jupyter-k8s-shared` namespace, where the templates live, rather than to the namespace your users work in. +`AmazonSagemakerHyperpodUserClusterPolicy` | See cluster-wide resources, which the Studio UI needs to render. Read access to namespaces and nodes, get access to custom resource definitions, read access to Kueue cluster queues, resource flavors, and workload priority classes, and permission to check the user’s own access. Attach it scoped to the cluster rather than to a namespace. @@ -116 +141 @@ Tag Key = “`grafana-workspace`”, Tag Value = “`ws-id`”. -You can restrict Kubernetes namespace permissions for users, so that they will only have access to view tasks belonging to a specified namespace. The following provides information on how to restrict the task view in Studio for EKS clusters. For information on viewable tasks in Studio, see [Tasks](./sagemaker-hyperpod-studio-tabs.html#sagemaker-hyperpod-studio-tabs-tasks). +Full access means get, list, watch, create, update, patch, and delete. @@ -118 +143 @@ You can restrict Kubernetes namespace permissions for users, so that they will o -Users will have visibility to all EKS cluster tasks by default. You can restrict users’ visibility for EKS cluster tasks to specified namespaces, ensuring that users can access the resources they need while maintaining strict access controls. You will need to provide the namespace for the user to display jobs of that namespace once the following is set up. +These are Amazon EKS cluster-access policies, not IAM managed policies. Their ARNs take the form `arn:<partition>:eks::aws:cluster-access-policy/<name>`. Attaching one through **Manage access** creates the Amazon EKS access entry for the execution role, which is what applies the policy to the role. @@ -120 +145 @@ Users will have visibility to all EKS cluster tasks by default. You can restrict -Once the restriction is applied, you will need to provide the namespace to the users assuming the role. Studio will only display the jobs of the namespace once the user provides inputs namespace they have permissions to view in the **Tasks** tab. +### Restrict with a custom Kubernetes RBAC role @@ -122 +147 @@ Once the restriction is applied, you will need to provide the namespace to the u -The following configuration allows administrators to grant specific, limited access to data scientists for viewing tasks within the cluster. This configuration grants the following permissions: +Use a custom role when you want to grant a custom set of permissions instead of the ones a cluster-access policy provides. The following configuration allows administrators to grant specific, limited access to data scientists for viewing tasks within the cluster. This configuration grants the following permissions: @@ -166 +191 @@ YAML Configuration - 2. Apply the configuration using [`kubectl`](https://kubernetes.io/docs/reference/kubectl/): + 2. Apply the configuration using [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) from the Kubernetes website: