AWS sagemaker documentation change
Summary
Expanded documentation on unallocated resource sharing in SageMaker HyperPod, including how it works, node eligibility criteria, and new borrow limit types (percentage-based and absolute). Added operational details about cluster monitoring and resource reconciliation.
Security assessment
The changes focus on operational aspects of resource sharing and cluster management. No security vulnerabilities, patches, or security-specific features are mentioned. Changes describe resource allocation mechanics and monitoring procedures without security implications.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md b/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md index f23d39aa1..13081004e 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-policies.md @@ -5 +5 @@ -Sharing idle compute resources examples +Sharing idle compute resources examplesHow unallocated resource sharing worksNode eligibility for unallocated resource sharing @@ -30,0 +31,2 @@ When data scientists submit jobs to the cluster, they use the priority class nam + * **Unallocated resource sharing** enables teams to borrow compute resources that are not allocated to any team through compute quota. When enabled, unallocated cluster capacity becomes available for teams to borrow automatically. For more information, see How unallocated resource sharing works. + @@ -48 +50,3 @@ The fair-share weight provides a comparative edge to this team when vying for av - * **Borrow limit** : The limit of idle compute that a team is allowed to borrow. A team can borrow up to 500% of allocated compute. The value you provide here is interpreted as a percentage. For example, a value of 500 will be interpreted as 500%. + * **Percentage-based borrow limit** : The limit of idle compute that a team is allowed to borrow, expressed as a percentage of their guaranteed quota. A team can borrow up to 10,000% of allocated compute. The value you provide here is interpreted as a percentage. For example, a value of 500 will be interpreted as 500%. This percentage applies uniformly across all resource types (CPU, GPU, Memory) and instance types in the team's quota. + + * **Absolute borrow limit** : The limit of idle compute that a team is allowed to borrow, defined as absolute resource values per instance type. This provides granular control over borrowing behavior for specific instance types. You need to specify absolute limits using the same schema as **Compute quota** , including instance count, accelerators, vCPU, memory, or accelerator partitions. You can specify absolute limits for one or more instance types in your team's quota. @@ -63 +67,65 @@ If any team's **Compute allocation** policy is set to **Don't Lend** , the team -To maintain a pool or a set of resources that all teams can borrow from, you can set up a dedicated team with resources that bridge the gap between other teams' allocations and the total cluster capacity. Ensure that this cumulative resource allocation includes the appropriate instance types and does not exceed the total cluster capacity. To ensure that these resources can be shared among teams, enable the participating teams to have their compute allocations set to **Lend and Borrow** or **Lend** for this common pool of resources. Every time new teams are introduced, quota allocations are changed, or there are any changes to the cluster capacity, revisit the quota allocations of all the teams and ensure the cumulative quota remains at or below cluster capacity. +## How unallocated resource sharing works + +Unallocated resource sharing automatically manages the pool of resources that are not allocated to any compute quota in your cluster. This means HyperPod continuously monitors your cluster state and automatically updates to the correct configuration over time. + +**Initial Setup** + + * When you set `IdleResourceSharing` to `Enabled` in your ClusterSchedulerConfig (by default it is `Disabled`), HyperPod task governance begins monitoring your cluster and calculates available idle resources by subtracting team quotas from total node capacity. + + * Unallocated resource sharing ClusterQueues are created to represent the borrowable resource pool. + + * When you first enable unallocated resource sharing, infrastructure setup takes several mins. You can monitor the progress through policy `Status` and `DetailedStatus` in ClusterSchedulerConfig. + + + + +**Ongoing Reconciliation** + + * HyperPod task governance continuously monitors for changes such as node additions or removals and cluster queue quota updates. + + * When changes occur, unallocated resource sharing recalculates quota and updates ClusterQueues. Reconciliation typically completes within seconds. + + + + +**Monitoring** + +You can verify that unallocated resource sharing is fully configured by checking for unallocated resource sharing ClusterQueues: + + + kubectl get clusterqueue | grep hyperpod-ns-idle-resource-sharing + +When you see ClusterQueues with names like `hyperpod-ns-idle-resource-sharing-cq-1`, unallocated resource sharing is active. Note that multiple unallocated resource sharing ClusterQueues may exist depending on the number of resource flavors in your cluster. + +## Node eligibility for unallocated resource sharing + +Unllocated Resource Sharing only includes nodes that meet the following requirements: + + 1. **Node Ready Status** + + * Nodes must be in `Ready` status to contribute to the unallocated resource pool. + + * Nodes in `NotReady` or other non-ready states are excluded from capacity calculations. + + * When a node becomes `Ready`, it is automatically included in the next reconciliation cycle. + + 2. **Node Schedulable Status** + + * Nodes with `spec.unschedulable: true` are excluded from unallocated resource sharing. + + * When a node becomes schedulable again, it is automatically included in the next reconciliation cycle. + + 3. **MIG Configuration (GPU nodes only)** + + * For GPU nodes with MIG (Multi-Instance GPU) partitioning, the `nvidia.com/mig.config.state` label must show `success` for the node to contribute MIG profiles to unallocated resource sharing. + + * These nodes will be retried automatically once MIG configuration completes successfully. + + 4. **Supported Instance Types** + + * The instance must be a supported SageMaker HyperPod instance type. + + * See the list of supported instance types in the SageMaker HyperPod cluster. + + +