AWS sagemaker documentation change
Summary
Updated documentation for configuring heterogeneous clusters in SageMaker training jobs. Added detailed examples for SageMaker Python SDK v3 while maintaining v2 examples. Changed estimator references to ModelTrainer and updated API links.
Security assessment
The changes consist of documentation updates for new SDK versions and configuration examples. No security vulnerabilities, patches, or security-specific features are mentioned. Changes include updated API references and code samples for data channel configuration, but without any security context or access control modifications.
Diff
diff --git a/sagemaker/latest/dg/train-heterogeneous-cluster-configure.md b/sagemaker/latest/dg/train-heterogeneous-cluster-configure.md index 8e26b05ac..bbebdf1a6 100644 --- a//sagemaker/latest/dg/train-heterogeneous-cluster-configure.md +++ b//sagemaker/latest/dg/train-heterogeneous-cluster-configure.md @@ -37 +37 @@ Follow instructions on how to configure instance groups for a heterogeneous clus - 1. To configure instance groups of a heterogeneous cluster for a training job, use the `sagemaker.instance_group.InstanceGroup` class. You can specify a custom name for each instance group, the instance type, and the number of instances for each instance group. For more information, see [sagemaker.instance_group.InstanceGroup](https://sagemaker.readthedocs.io/en/stable/api/utility/instance_group.html) in the _SageMaker AI Python SDK documentation_. + 1. To configure instance groups of a heterogeneous cluster for a training job, use the `sagemaker.instance_group.InstanceGroup` class. You can specify a custom name for each instance group, the instance type, and the number of instances for each instance group. For more information, see [sagemaker.instance_group.InstanceGroup](https://sagemaker.readthedocs.io/en/stable/api/sagemaker_core.html) in the _SageMaker AI Python SDK documentation_. @@ -58 +58,43 @@ The preceding diagram shows a conceptual example of how pre-training processes, - 2. Using the instance group objects, set up training input channels and assign instance groups to the channels through the `instance_group_names` argument of the [sagemaker.inputs.TrainingInput](https://sagemaker.readthedocs.io/en/stable/api/utility/inputs.html) class. The `instance_group_names` argument accepts a list of strings of instance group names. + 2. Using the instance group objects, set up training input channels and assign instance groups to the channels through the `instance_group_names` argument. The `instance_group_names` argument accepts a list of strings of instance group names. + +The following example shows how to set two training input channels and assign the instance groups created in the example of the previous step. You can also specify Amazon S3 bucket paths for the instance groups to process data for your usage purposes. + +SageMaker Python SDK v3 + + + from sagemaker.train.configs import InputData + from sagemaker.core.shapes import Channel, DataSource, S3DataSource + + training_input_channel_1 = Channel( + channel_name='training', + data_source=DataSource( + s3_data_source=S3DataSource( + s3_data_type='S3Prefix', + s3_uri='s3://your-training-data-storage/folder1', + s3_data_distribution_type='FullyReplicated', # Available Options: FullyReplicated | ShardedByS3Key + instance_group_names=["instance_group_1"], + ) + ), + input_mode='File', # Available Options: File | Pipe | FastFile + ) + + training_input_channel_2 = Channel( + channel_name='dummy-input-channel', + data_source=DataSource( + s3_data_source=S3DataSource( + s3_data_type='S3Prefix', + s3_uri='s3://your-training-data-storage/folder2', + s3_data_distribution_type='FullyReplicated', + instance_group_names=["instance_group_2"], + ) + ), + input_mode='File', + ) + +For more information about the arguments of `InputData`, see the following links. + + * The [sagemaker.train.configs.InputData](https://sagemaker.readthedocs.io/en/stable/api/sagemaker_train.html) class in the _SageMaker Python SDK documentation_ + + * The [S3DataSource](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_S3DataSource.html) API in the _SageMaker AI API Reference_ + +SageMaker Python SDK v2 (Legacy) @@ -60 +101,0 @@ The preceding diagram shows a conceptual example of how pre-training processes, -The following example shows how to set two training input channels and assign the instance groups created in the example of the previous step. You can also specify Amazon S3 bucket paths to the `s3_data` argument for the instance groups to process data for your usage purposes. @@ -86 +127 @@ For more information about the arguments of `TrainingInput`, see the following l - 3. Configure a SageMaker AI estimator with the `instance_groups` argument as shown in the following code example. The `instance_groups` argument accepts a list of `InstanceGroup` objects. + 3. Configure a SageMaker AI training object with the `instance_groups` argument as shown in the following code example. The `instance_groups` argument accepts a list of `InstanceGroup` objects. @@ -90 +131,3 @@ For more information about the arguments of `TrainingInput`, see the following l -The heterogeneous cluster feature is available through the SageMaker AI [PyTorch](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html) and [TensorFlow](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator) framework estimator classes. Supported frameworks are PyTorch v1.10 or later and TensorFlow v2.6 or later. To find a complete list of available framework containers, framework versions, and Python versions, see [SageMaker AI Framework Containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#sagemaker-framework-containers-sm-support-only) in the AWS Deep Learning Container GitHub repository. +The heterogeneous cluster feature is available through the SageMaker AI [PyTorch](https://sagemaker.readthedocs.io/en/stable/api/sagemaker_train.html) and [TensorFlow](https://sagemaker.readthedocs.io/en/stable/api/sagemaker_train.html) ModelTrainer classes. Supported frameworks are PyTorch v1.10 or later and TensorFlow v2.6 or later. To find a complete list of available framework containers, framework versions, and Python versions, see [SageMaker AI Framework Containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#sagemaker-framework-containers-sm-support-only) in the AWS Deep Learning Container GitHub repository. + +SageMaker Python SDK v3 @@ -92 +134,0 @@ The heterogeneous cluster feature is available through the SageMaker AI [PyTorch -PyTorch @@ -93,0 +136,44 @@ PyTorch +**PyTorch** + + from sagemaker.train import ModelTrainer + from sagemaker.train.configs import SourceCode + from sagemaker.core import image_uris + + training_image = image_uris.retrieve( + framework="pytorch", region=region, version='x.y.z', # 1.10.0 or later + py_version='pyxy', instance_type='ml.p3dn.24xlarge', + image_scope="training" + ) + + model_trainer = ModelTrainer( + ... + source_code=SourceCode(entry_script='my-training-script.py'), + training_image=training_image, + job_name='my-training-job-with-heterogeneous-cluster', + instance_groups=[instance_group_1, instance_group_2] + ) + +**TensorFlow** + + from sagemaker.train import ModelTrainer + from sagemaker.train.configs import SourceCode + from sagemaker.core import image_uris + + training_image = image_uris.retrieve( + framework="tensorflow", region=region, version='x.y.z', # 2.6.0 or later + py_version='pyxy', instance_type='ml.p3dn.24xlarge', + image_scope="training" + ) + + model_trainer = ModelTrainer( + ... + source_code=SourceCode(entry_script='my-training-script.py'), + training_image=training_image, + job_name='my-training-job-with-heterogeneous-cluster', + instance_groups=[instance_group_1, instance_group_2] + ) + +SageMaker Python SDK v2 (Legacy) + + +**PyTorch** @@ -106,2 +192 @@ PyTorch -TensorFlow - +**TensorFlow** @@ -122 +207 @@ TensorFlow -The `instance_type` and `instance_count` argument pair and the `instance_groups` argument of the SageMaker AI estimator class are mutually exclusive. For homogeneous cluster training, use the `instance_type` and `instance_count` argument pair. For heterogeneous cluster training, use `instance_groups`. +The `instance_type` and `instance_count` argument pair and the `instance_groups` argument of the SageMaker AI ModelTrainer class are mutually exclusive. For homogeneous cluster training, use the `instance_type` and `instance_count` argument pair. For heterogeneous cluster training, use `instance_groups`. @@ -128 +213,14 @@ To find a complete list of available framework containers, framework versions, a - 4. Configure the `estimator.fit` method with the training input channels configured with the instance groups and start the training job. + 4. Start the training job with the training input channels configured with the instance groups. + +SageMaker Python SDK v3 + + + model_trainer.train( + inputs={ + 'training': training_input_channel_1, + 'dummy-input-channel': training_input_channel_2 + } + ) + +SageMaker Python SDK v2 (Legacy) +