AWS sagemaker documentation change
Summary
Added a comprehensive 'Workload specification reference' section with complete JSON schema documentation, updated terminology from 'traffic patterns' to 'request rate' in benchmarking parameters, and added navigation reference.
Security assessment
The change introduces a 'secrets' section that explicitly recommends using AWS Secrets Manager for sensitive values (like Hugging Face tokens) instead of plaintext in parameters, which is a security best practice. However, there is no evidence this addresses a specific security vulnerability or incident; it appears to be routine documentation enhancement for security features.
Diff
diff --git a/sagemaker/latest/dg/generative-ai-inference-recommendations-workload-config.md b/sagemaker/latest/dg/generative-ai-inference-recommendations-workload-config.md index c0c8feb1b..b61e9d5f0 100644 --- a//sagemaker/latest/dg/generative-ai-inference-recommendations-workload-config.md +++ b//sagemaker/latest/dg/generative-ai-inference-recommendations-workload-config.md @@ -7 +7 @@ -Create a workload configuration with inline parametersCreate a workload configuration with a datasetWorkload configuration for benchmarkingManage workload configurations +Create a workload configuration with inline parametersCreate a workload configuration with a datasetWorkload configuration for benchmarkingManage workload configurationsWorkload specification reference @@ -91 +91 @@ By default, synthetic prompts are generated. You can also use a public dataset o -When creating a workload configuration for benchmarking an existing endpoint, you can specify additional parameters such as the tokenizer, concurrency, request count, and traffic patterns. +When creating a workload configuration for benchmarking an existing endpoint, you can specify additional parameters such as the tokenizer, concurrency, request count, and request rate. @@ -105,5 +105,2 @@ When creating a workload configuration for benchmarking an existing endpoint, yo - }, - "traffic_pattern": { - "requests_per_second": 1, - "concurrency": 1, - "duration_seconds": 60, + "request_rate": 1.0, + "benchmark_duration": 60, @@ -135,0 +133,352 @@ Use the following operations to manage your workload configurations. +## Workload specification reference + +This section provides the complete schema for the workload specification JSON document that you pass in the `WorkloadSpec.Inline` field when creating a workload configuration. + +### Syntax + +The following is a representative example of a workload specification with commonly used parameters. All parameters are documented in the reference table below. + + + { + "benchmark": { + "type": "aiperf" + }, + "parameters": { + "prompt_input_tokens_mean": 550, + "prompt_input_tokens_stddev": 150.0, + "output_tokens_mean": 150, + "output_tokens_stddev": 50.0, + "concurrency": 10, + "request_count": 100, + "request_rate": 5.0, + "benchmark_duration": 120, + "streaming": true, + "tokenizer": "meta-llama/Llama-3.2-1B" + }, + "secrets": { + "hf_token": "arn:aws:secretsmanager:us-west-2:111122223333:secret:my-hf-token-AbCdEf" + }, + "tooling": { + "api_standard": "openai" + } + } + + +### Workload specification keys + +The workload specification contains the following top-level keys. Unknown keys are rejected. + +#### benchmark + +Required mapping. Identifies the benchmarking tool to use. + +`benchmark/type` + + +Required. The benchmark engine. The only valid value is `aiperf`. + +#### parameters + +Optional mapping. Benchmark parameters passed to the AIPerf engine. Unknown parameter names are rejected. All parameters are optional unless noted otherwise. + +**Token distribution** + +`prompt_input_tokens_mean` + + +Integer. Mean number of input tokens per request for synthetic prompt generation. Aliases: `synthetic_input_tokens_mean`, `isl`. + +`prompt_input_tokens_stddev` + + +Float. Standard deviation of input token count. Aliases: `synthetic_input_tokens_stddev`, `isl_stddev`. + +`output_tokens_mean` + + +Integer. Mean number of output tokens per request. Aliases: `prompt_output_tokens_mean`, `osl`. + +`output_tokens_stddev` + + +Float. Standard deviation of output token count. Aliases: `prompt_output_tokens_stddev`, `osl_stddev`. + +**Traffic shaping** + +`concurrency` + + +Integer. Number of concurrent requests to send during the benchmark. + +`request_count` + + +Integer. Total number of requests to send. Alias: `num_requests`. + +`request_rate` + + +Float. Target requests per second. + +`benchmark_duration` + + +Integer. Duration of the benchmark in seconds. + +`max_concurrency` + + +Integer. Maximum number of concurrent requests allowed. + +`request_rate_mode` + + +String. Request arrival pattern. Alias: `arrival_pattern`. + +`arrival_smoothness` + + +Float. Controls burstiness of request arrivals. Higher values produce smoother traffic. Alias: `vllm_burstiness`. + +`prefill_concurrency` + + +Integer. Number of concurrent prefill requests. + +**General** + +`streaming` + + +Boolean. Whether to use streaming responses. Default: `true`. + +`tokenizer` + + +String. HuggingFace model name or local directory path for the tokenizer used to count tokens. Example: `meta-llama/Llama-3.2-1B`. + +`hf_token` + + +String. Hugging Face access token for downloading gated models and tokenizers. Alias: `HF_TOKEN`. For sensitive values, use the `secrets` section instead of passing the token in plaintext. + +`request_timeout_seconds` + + +Integer. Timeout in seconds for individual requests. + +`benchmark_grace_period` + + +Integer. Grace period in seconds after the benchmark completes to allow in-flight requests to finish. + +`extra_inputs` + + +String. Additional JSON-encoded inputs to include in each request payload. + +`random_seed` + + +Integer. Seed for random number generation. Default: `42`. + +`verbose` + + +Boolean. Enable verbose logging. Default: `false`. + +`num_conversations` + + +Integer. Number of multi-turn conversations to simulate. Aliases: `conversation_num`, `num_sessions`. + +`model_selection_strategy` + + +String. Strategy for selecting models when multiple models are available on the endpoint. + +**Warmup** + +Warmup parameters control an optional warm-up phase that runs before the measured benchmark. This primes the model server's caches and JIT compilation. + +`warmup_duration` + + +Integer. Duration of the warmup phase in seconds. + +`warmup_request_count` + + +Integer. Number of warmup requests. Alias: `num_warmup_requests`. +