AWS sagemaker documentation change
Summary
Updated Nova model evaluation documentation with new benchmarks (rubric_llm_judge, aime_2024, calendar_scheduling, humaneval), added MLflow tracking parameters, modified model configurations, introduced reasoning_effort parameter, updated postprocessor configuration, expanded instance count ranges, and replaced troubleshooting content.
Security assessment
The changes focus on feature enhancements (new benchmarks, MLflow integration) and configuration updates. No evidence of security vulnerability fixes, security incident responses, or new security controls. Changes like added lambda_type options and instance count modifications are operational improvements without security implications.
Diff
diff --git a/sagemaker/latest/dg/nova-model-evaluation.md b/sagemaker/latest/dg/nova-model-evaluation.md index a4675b197..6f4929154 100644 --- a//sagemaker/latest/dg/nova-model-evaluation.md +++ b//sagemaker/latest/dg/nova-model-evaluation.md @@ -72,2 +72 @@ ifeval | Text | Instruction-Following Evaluation – Gauges how accurately a mo -gen_qa | Multi-Modal (image) | Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. `gen_qa` supports image inference for Amazon Nova Lite or Amazon Nova Pro based models. | all | gen_qa | No -mmmu | Multi-Modal | Massive Multidiscipline Multimodal Understanding (MMMU) – College-level benchmark comprising multiple-choice and open-ended questions from 30 disciplines. | accuracy | zs_cot | Yes +gen_qa | Multi-Modal (image) | Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. `gen_qa` supports image inference for Amazon Nova Lite or Amazon Nova Pro based models. Also supports Bring-Your-Own Metrics lambda. (For RFT evaluation, please use RFT eval recipe) | all | gen_qa | No @@ -75,0 +75,4 @@ mm_llm_judge | Multi-Modal (image) | This new benchmark behaves the same as the +rubric_llm_judge | Text | Rubric Judge is an enhanced LLM-as-a-judge evaluation model built on Nova 2.0 Lite. Unlike the [original judge model](https://aws.amazon.com/blogs/machine-learning/evaluating-generative-ai-models-with-amazon-nova-llm-as-a-judge-on-amazon-sagemaker-ai/) that only provides preference verdicts, Rubric Judge dynamically generates custom evaluation criteria tailored to each prompt and assigns granular scores across multiple dimensions. | all | judge | No +aime_2024 | Text | AIME 2024 - American Invitational Mathematics Examination problems testing advanced mathematical reasoning and problem-solving | exact_match | zs_cot | No +calendar_scheduling | Text | Natural Plan - Calendar Scheduling task testing planning abilities for scheduling meetings across multiple days and people | exact_match | fs | No +humaneval | Text | HumanEval - A benchmark dataset designed to evaluate the code generation capabilities of large language models | pass@1 | zs | No @@ -88,2 +91,2 @@ Below is a breakdown of the key components in the recipe and guidance on how to - model_type: amazon.nova-micro-v1:0:128k - model_name_or_path: nova-micro/prod + model_type: amazon.nova-2-lite-v1:0:256k + model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location @@ -91,0 +95,3 @@ Below is a breakdown of the key components in the recipe and guidance on how to + mlflow_tracking_uri: "" + mlflow_experiment_name : "" + mlflow_run_name : "" @@ -102,0 +109,2 @@ Below is a breakdown of the key components in the recipe and guidance on how to + * amazon.nova-2-lite-v1:0:256k + @@ -110,0 +119,2 @@ Below is a breakdown of the key components in the recipe and guidance on how to + * nova-lite-2/prod + @@ -122,0 +133,6 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste + * `mlflow_tracking_uri`: (Optional) The location of the MLflow tracking server (only needed on SMHP) + + * `mlflow_experiment_name`: (Optional) Name of the experiment to group related ML runs together + + * `mlflow_run_name`: (Optional) Custom name for a specific training run within an experiment + @@ -153,2 +168,0 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste - * `mmmu` - @@ -158,0 +173,8 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste + * `rubric_llm_judge` + + * `aime_2024` + + * `calendar_scheduling` + + * `humaneval` + @@ -244,0 +267 @@ For `llm_judge` and `mm_llm_judge`, bring your own dataset benchmark, return fol + reasoning_effort: null # options: low/high to enable reasoning or null to disable reasoning @@ -255,0 +279 @@ For `llm_judge` and `mm_llm_judge`, bring your own dataset benchmark, return fol + * `reasoning_effort`: controls the reasoning behavior for reasoning-capable models. Set `reasoning_effort` only when `model_type` specifies a reasoning-capable model (currently `amazon.nova-2-lite-v1:0:256k`). Available options are null (default value if not set; disables reasoning), low, or high. @@ -259,58 +282,0 @@ For `llm_judge` and `mm_llm_judge`, bring your own dataset benchmark, return fol -### Reasoning model support for evaluation - -Amazon Nova Lite 2.0 (amazon.nova-2-lite-v1:0:256k) supports an explicit reasoning mode that enables the model to perform internal reasoning steps before generating final responses. This capability is controlled through the `reasoning_effort` parameter in your evaluation recipe. - - - run: - name: "nova-evaluation-with-reasoning" - model_type: "amazon.nova-2-lite-v1:0:256k" - model_name_or_path: "nova-lite-2/prod" - replicas: 1 - data_s3_path: "s3://your-bucket/evaluation-data.jsonl" - output_s3_path: "s3://your-bucket/evaluation-results/" - - evaluation: - task: "mmlu" - strategy: "generate" - metric: "all" - - inference: - reasoning_effort: "high" - max_new_tokens: 200 - top_k: 50 - top_p: 1.0 - temperature: 0 - -**Enable reasoning (low or high) for:** - - * Mathematical problem-solving - - * Multi-step logical deductions - - * Code generation and debugging - - * Complex analytical questions requiring intermediate steps - - * Tasks where showing your work improves accuracy - - - - -**Use non-reasoning mode (omit parameter) for:** - - * Simple Q&A or factual queries - - * Creative writing tasks - - * Classification tasks - - * When faster response times are critical - - * Performance benchmarking where reasoning overhead should be excluded - - - - -###### Important - -Only `amazon.nova-2-lite-v1:0:256k` currently supports reasoning mode. Using `reasoning_effort` with unsupported models will fail with a ConfigValidationError. @@ -331,11 +296,0 @@ Recipe format: `xxx_general_text_benchmark_eval.yaml`. -These recipes enable you to evaluate the fundamental capabilities of Amazon Nova models across a comprehensive suite of multi-modality benchmarks. - -Recipe format: `xxx_general_multi_modal_benchmark_eval.yaml`. - -**Multi-modal benchmark requirements** - - * Model support - Only support nova-lite and nova-pro base model and its post-trained variants. - - - - @@ -577,0 +533 @@ Postprocessor | + lambda_type: "custom_metrics" @@ -586 +542 @@ Postprocessor | - * `preprocessing`: Whether to enable custom pre-processing operations + * `lambda_type`: "custom_metrics" or "rft". @@ -588 +544 @@ Postprocessor | - * `postprocessing`: Whether to enable custom post-processing operations + * `preprocessing`: Whether to enable custom pre-processing operations. @@ -590 +546,3 @@ Postprocessor | - * `aggregation`: Built-in aggregation function (valid options: min, max, average, sum) + * `postprocessing`: Whether to enable custom post-processing operations. + + * `aggregation`: Built-in aggregation function (valid options: min, max, average, sum). @@ -768 +726 @@ To use your custom dataset, modify your evaluation recipe with the following req -Start a training job using the following sample Jupyter notebook. For more information, see [Use a SageMaker AI estimator to run a training job](https://docs.aws.amazon.com//sagemaker/latest/dg/docker-containers-adapt-your-own-private-registry-estimator.html). +Start a training job using the following sample Jupyter notebook. Please refer to below notebook as example to run the evaluation training job. For more information, see [Use a SageMaker AI estimator to run a training job](https://docs.aws.amazon.com//sagemaker/latest/dg/docker-containers-adapt-your-own-private-registry-estimator.html). @@ -784,3 +742,3 @@ Model | Job type | Instance type | Recommended instance count | Allowed instance -Amazon Nova Micro | Evaluation (SFT/DPO) | g5.12xlarge | 1 | 1 -Amazon Nova Lite | Evaluation (SFT/DPO) | g5.12xlarge | 1 | 1 -Amazon Nova Pro | Evaluation (SFT/DPO) | p5.48xlarge | 1 | 1 +Amazon Nova Micro | Evaluation (SFT/DPO) | g5.12xlarge | 1 | 1 - 16 +Amazon Nova Lite | Evaluation (SFT/DPO) | g5.12xlarge | 1 | 1 - 16 +Amazon Nova Pro | Evaluation (SFT/DPO) | p5.48xlarge | 1 | 1 - 16 @@ -793,0 +752,3 @@ The following sample notebook demonstrates how to run an evaluation training job + + # Do not use sagemaker v3, as sagemaker v3 introduced breaking changes + @@ -945 +906 @@ You can use CloudWatch log group `/aws/sagemaker/TrainingJobs` for training job -#### CUDA Out of Memory Error +#### Engine core Failure @@ -949 +910 @@ You can use CloudWatch log group `/aws/sagemaker/TrainingJobs` for training job -When running model evaluation, you receive the following error: +If you are seeing: @@ -952,2 +913 @@ When running model evaluation, you receive the following error: - torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X MiB. - GPU 0 has a total capacity of Y GiB of which Z MiB is free. + RuntimeError: Engine core initialization failed. @@ -957,9 +917 @@ When running model evaluation, you receive the following error: -This error occurs when you attempt to load a model that requires more GPU memory than what's available on your current instance type. - -**Solution** : - -Choose an instance type with more GPU memory. For example, if you use G5.12xlarge (96 GiB GPU memory), upgrade to G5.48xlarge (192 GiB GPU memory) - -**Prevention** : - -Before running model evaluation, do the following. +Although this is a general error that can have multiple causes, it typically occurs when there is a mismatch between the model checkpoint you're trying to load and the model type specified. E.g. you want to evaluate a fine-tuned Nova 2.0 lite model checkpoint but the model type you provide is 1.0 model type. e.g. `amazon.nova-micro-v1:0:128k` @@ -967 +919 @@ Before running model evaluation, do the following. - * Estimate your model's memory requirements +The correct mapping should be @@ -969,3 +920,0 @@ Before running model evaluation, do the following. - * Ensure your selected instance type has sufficient GPU memory - - * Consider the memory overhead needed for model loading and inference @@ -972,0 +922,2 @@ Before running model evaluation, do the following. + model_type: amazon.nova-2-lite-v1:0:256k + model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location @@ -973,0 +925 @@ Before running model evaluation, do the following. +**Prevention** : @@ -974,0 +927 @@ Before running model evaluation, do the following. +Double check the `model_name_or_path` is mapped to the right `model_type` before submitting the evaluation job. @@ -978 +931 @@ Before running model evaluation, do the following. -The following lists available subtasks for model evaluation across multiple domains including MMLU (Massive Multitask Language Understanding), BBH (Big Bench Hard), mathematics, and MMMU (Massive Multi-discipline Multimodal Understanding). These subtasks allow you to assess your model's performance on specific capabilities and knowledge areas. +The following lists available subtasks for model evaluation across multiple domains including MMLU (Massive Multitask Language Understanding), BBH (Big Bench Hard), and MATH. These subtasks allow you to assess your model's performance on specific capabilities and knowledge areas. @@ -1087,37 +1039,0 @@ The following lists available subtasks for model evaluation across multiple doma - ] - -**MMMU** - - - MATH_SUBTASKS = [ - "Accounting", - "Agriculture", - "Architecture_and_Engineering", - "Art", - "Art_Theory", - "Basic_Medical_Science",