AWS sagemaker documentation change
Summary
Entire documentation content removed and replaced with a redirect notice to Nova 1.0/2.0 user guides
Security assessment
The change removes all technical content about model evaluation processes without introducing any new security information. This appears to be documentation reorganization/relocation rather than addressing security vulnerabilities or adding security features. No security-related content was modified or added in the remaining text.
Diff
diff --git a/sagemaker/latest/dg/nova-model-evaluation.md b/sagemaker/latest/dg/nova-model-evaluation.md index 939638486..1d30bed8e 100644 --- a//sagemaker/latest/dg/nova-model-evaluation.md +++ b//sagemaker/latest/dg/nova-model-evaluation.md @@ -5,2 +4,0 @@ -PrerequisitesAvailable benchmark tasksEvaluation specific configurationsEvaluation training jobsAssessing evaluation resultsBest practices and troubleshootingAvailable subtasks - @@ -9,1050 +7 @@ PrerequisitesAvailable benchmark tasksEvaluation specific configurationsEvaluati -The purpose of the evaluation process is to assess trained-model performance against benchmarks or custom dataset. The evaluation process typically involves steps to create evaluation recipe pointing to the trained model, specify evaluation datasets and metrics, submit a separate job for the evaluation, and evaluate against standard benchmarks or custom data. The evaluation process will output performance metrics stored in your Amazon S3 bucket. - -###### Note - -The evaluation process described in this topic is an offline process. The model is tested against fixed benchmarks with predefined answers, rather than being assessed in real-time or through live user interactions. For real-time evaluation, you can test the model after it has been deployed to Amazon Bedrock by calling [Amazon Bedrock](https://docs.aws.amazon.com//bedrock/latest/userguide/import-with-create-custom-model.html) Runtime APIs. - -###### Topics - - * Prerequisites - - * Available benchmark tasks - - * Evaluation specific configurations - - * Running evaluation training jobs - - * Assessing and analyzing evaluation results - - * Evaluation best practices and troubleshooting - - * Available subtasks - - * [Rubric Based Judge](./nova-rubric-judge-evaluation.html) - - * [Reasoning model evaluation](./nova-reasoning-model-evaluation.html) - - * [RFT evaluation](./nova-rft-evaluation.html) - - * [Implementing reward functions](./nova-implementing-reward-functions.html) - - * [Running evaluations and interpreting results](./nova-running-evaluations.html) - - - - -## Prerequisites - -Before you start a evaluation training job, note the following. - - * A SageMaker AI-trained Amazon Nova model which you want to evaluate its performance. - - * Base Amazon Nova recipe for evaluation. For more information, see [Getting Amazon Nova recipes](./nova-model-recipes.html#nova-model-get-recipes). - - - - -## Available benchmark tasks - -A sample code package is available that demonstrates how to calculate benchmark metrics using the SageMaker model evaluation feature for Amazon Nova. To access the code packages, see [sample-Nova-lighteval-custom-task](https://github.com/aws-samples/sample-Nova-lighteval-custom-task/). - -Here is a list of available industry standard benchmarks supported. You can specify the following benchmarks in the `eval_task` parameter. - -**Available benchmarks for model evaluation** - -Benchmark | Modality | Description | Metrics | Strategy | Subtask available ----|---|---|---|---|--- -mmlu | Text | Multi-task Language Understanding – Tests knowledge across 57 subjects. | accuracy | zs_cot | Yes -mmlu_pro | Text | MMLU – Professional Subset – Focuses on professional domains such as law, medicine, accounting, and engineering. | accuracy | zs_cot | No -bbh | Text | Advanced Reasoning Tasks – A collection of challenging problems that test higher-level cognitive and problem-solving skills. | accuracy | fs_cot | Yes -gpqa | Text | General Physics Question Answering – Assesses comprehension of physics concepts and related problem-solving abilities. | accuracy | zs_cot | No -math | Text | Mathematical Problem Solving – Measures mathematical reasoning across topics including algebra, calculus, and word problems. | exact_match | zs_cot | Yes -strong_reject | Text | Quality-Control Task – Tests the model’s ability to detect and reject inappropriate, harmful, or incorrect content. | deflection | zs | Yes -ifeval | Text | Instruction-Following Evaluation – Gauges how accurately a model follows given instructions and completes tasks to specification. | accuracy | zs | No -gen_qa | Multi-Modal (image) | Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. `gen_qa` supports image inference for Amazon Nova Lite or Amazon Nova Pro based models. Also supports Bring-Your-Own Metrics lambda. (For RFT evaluation, please use RFT eval recipe) | all | gen_qa | No -llm_judge | Text | LLM-as-a-Judge Preference Comparison – Uses a Nova Judge model to determine preference between paired responses (B compared with A) for your prompts, calculating the probability of B being preferred over A. | all | judge | No -mm_llm_judge | Multi-Modal (image) | This new benchmark behaves the same as the text-based `llm_judge`above. The only difference is that it supports image inference. | all | judge | No -rubric_llm_judge | Text | Rubric Judge is an enhanced LLM-as-a-judge evaluation model built on Nova 2.0 Lite. Unlike the [original judge model](https://aws.amazon.com/blogs/machine-learning/evaluating-generative-ai-models-with-amazon-nova-llm-as-a-judge-on-amazon-sagemaker-ai/) that only provides preference verdicts, Rubric Judge dynamically generates custom evaluation criteria tailored to each prompt and assigns granular scores across multiple dimensions. | all | judge | No -aime_2024 | Text | AIME 2024 - American Invitational Mathematics Examination problems testing advanced mathematical reasoning and problem-solving | exact_match | zs_cot | No -calendar_scheduling | Text | Natural Plan - Calendar Scheduling task testing planning abilities for scheduling meetings across multiple days and people | exact_match | fs | No -humaneval | Text | HumanEval - A benchmark dataset designed to evaluate the code generation capabilities of large language models | pass@1 | zs | No - -## Evaluation specific configurations - -Below is a breakdown of the key components in the recipe and guidance on how to modify them for your use cases. - -### Understanding and modifying your recipes - -**General run configuration** - - - run: - name: eval_job_name - model_type: amazon.nova-2-lite-v1:0:256k - model_name_or_path: nova-lite-2/prod # or s3://escrow_bucket/model_location - replicas: 1 - data_s3_path: "" - mlflow_tracking_uri: "" - mlflow_experiment_name : "" - mlflow_run_name : "" - - * `name`: A descriptive name for your evaluation job. - - * `model_type`: Specifies the Nova model variant to use. Do not manually modify this field. Options include: - - * amazon.nova-micro-v1:0:128k - - * amazon.nova-lite-v1:0:300k - - * amazon.nova-pro-v1:0:300k - - * amazon.nova-2-lite-v1:0:256k - - * `model_name_or_path`: The path to the base model or s3 path for post trained checkpoint. Options include: - - * nova-micro/prod - - * nova-lite/prod - - * nova-pro/prod - - * nova-lite-2/prod - - * S3 path for post trained checkpoint path (`s3:customer-escrow-111122223333-smtj-<unique_id>/<training_run_name>`) - -###### Note - -**Evaluate post-trained model** - -To evaluate a post-trained model after a Nova SFT training job, follow these steps after running a successful training job. At the end of the training logs, you will see the log message "Training is complete". You will also find a `manifest.json` file in your output bucket containing the location of your checkpoint. This file will be located within an `output.tar.gz` file at your output S3 location. To proceed with evaluation, use this checkpoint by setting it as the value for `run.model_name_or_path` in your recipe configuration. - - * `replica`: The number of compute instances to use for distributed inference (running inference across multiple nodes). Set `replica` > 1 to enable multi-node inference, which accelerates evaluation. If both `instance_count` and `replica` are specified, `instance_count` takes precedence. Note that multiple replicas only apply to SageMaker training jobs, not SageMaker HyperPod. - - * `data_s3_path`: The input dataset Amazon S3 path. This field is required but should always left empty. - - * `mlflow_tracking_uri`: (Optional) The location of the MLflow tracking server (only needed on SMHP) - - * `mlflow_experiment_name`: (Optional) Name of the experiment to group related ML runs together - - * `mlflow_run_name`: (Optional) Custom name for a specific training run within an experiment - - - - -**Evaluation configuration** - - - evaluation: - task: mmlu - strategy: zs_cot - subtask: abstract_algebra - metric: accuracy - - * `task`: Specifies the evaluation benchmark or task to use. Supported task includes: - - * `mmlu` - - * `mmlu_pro` - - * `bbh` - - * `gpqa` - - * `math` - - * `strong_reject` - - * `gen_qa` - - * `ifeval` - - * `llm_judge` - - * `mm_llm_judge` - - * `rubric_llm_judge` - - * `aime_2024` - - * `calendar_scheduling` - - * `humaneval` - - * `strategy`: Defines the evaluation approach. - - * `zs_cot`: Zero-shot Chain of Thought - an approach to prompt large language models that encourages step-by-step reasoning without requiring explicit examples. - - * `fs_cot`: Few-shot Chain of Thought - an approach that provides a few examples of step-by-step reasoning before asking the model to solve a new problem. - - * `zs`: Zero-shot - an approach to solve a problem without any prior training examples. - - * `gen_qa`: Strategy specific for bring your own dataset. - - * `judge`: Strategy specific for Nova LLM as Judge and `mm_llm_judge`. - - * `subtask`: Optional. Specific components of the evaluation task. For a complete list of available subtasks, see Available subtasks. - - * Check supported subtasks in Available benchmarks tasks. - - * Should remove this field if there are no subtasks benchmarks. - - * `metric`: The evaluation metric to use. -