AWS sagemaker documentation change
Summary
Expanded documentation for Amazon Nova model evaluation jobs including benchmark tasks, recipe parameters, evaluation types, job setup instructions, and result visualization details
Security assessment
Added documentation for 'strong_reject' benchmark which tests model's ability to detect harmful/inappropriate content, and mentions 'computer_security' as a subtask. However, these are documented security features rather than fixes for existing vulnerabilities.
Diff
diff --git a/sagemaker/latest/dg/nova-hp-evaluate.md b/sagemaker/latest/dg/nova-hp-evaluate.md index 17ebbde85..fa3c6040f 100644 --- a//sagemaker/latest/dg/nova-hp-evaluate.md +++ b//sagemaker/latest/dg/nova-hp-evaluate.md @@ -4,0 +5,2 @@ +Available benchmark tasksUnderstanding the recipe parametersEvaluation recipe examplesStart an evaluation jobAccess and visualize your results + @@ -7 +9,577 @@ -An evaluation recipe is a YAML configuration file that defines how Amazon Nova model evaluation jobs are executed. This process allows you to assess the performance of either base or customized models against standard benchmarks or your own custom datasets. The evaluation generates quantitative metrics that help determine if additional model customization is needed, with results that can be stored in Amazon S3 or visualized in TensorBoard. +An evaluation recipe is a YAML configuration file that defines how your Amazon Nova model evaluation job is executed. With this recipe, you can assess the performance of a base or trained model against common benchmarks or your own custom datasets. Metrics can be stored in Amazon S3 or TensorBoard. The evaluation provides quantitative metrics that help you assess model performance across various tasks to determine if further customization is needed. + +Model evaluation is an offline process, where models are tested against fixed benchmarks with predefined answers. They are not assessed in real-time or against live user interactions. For real-time evaluations, you can evaluate the model after it is deployed to Amazon Bedrock by calling the Amazon Bedrock runtime APIs. + +###### Topics + + * Available benchmark tasks + + * Understanding the recipe parameters + + * Evaluation recipe examples + + * Start an evaluation job + + * Access and visualize your results + + + + +## Available benchmark tasks + +A sample code package is available that demonstrates how to calculate benchmark metrics using the SageMaker AI model evaluation feature for Amazon Nova. To access the code packages, see [sample-Nova-lighteval-custom-task](https://github.com/aws-samples/sample-Nova-lighteval-custom-task/). + +Here is a list of the supported, available industry standard benchmarks. You can specify the following benchmarks in the `eval_task` parameter: + +Benchmark | Modality | Description | Metrics | Strategy | Subtask Available +---|---|---|---|---|--- +mmlu | Text | Multi-task Language Understanding – Tests knowledge across 57 subjects. | accuracy | zs_cot | Yes +mmlu_pro | Text | MMLU – Professional Subset – Focuses on professional domains such as law, medicine, accounting, and engineering. | accuracy | zs_cot | No +bbh | Text | Advanced Reasoning Tasks – A collection of challenging problems that test higher-level cognitive and problem-solving skills. | accuracy | zs_cot | Yes +gpqa | Text | General Physics Question Answering – Assesses comprehension of physics concepts and related problem-solving abilities. | accuracy | zs_cot | No +math | Text | Mathematical Problem Solving – Measures mathematical reasoning across topics including algebra, calculus, and word problems. | exact_match | zs_cot | Yes +strong_reject | Text | Quality-Control Task – Tests the model’s ability to detect and reject inappropriate, harmful, or incorrect content. | deflection | zs | Yes +IFEval | Text | Instruction-Following Evaluation – Gauges how accurately a model follows given instructions and completes tasks to specification. | accuracy | zs | No +gen_qa | Text | Custom Dataset Evaluation – Lets you bring your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. | all | gen_qa | No +mmmu | Multi-modal | Massive Multidiscipline Multimodal Understanding (MMMU) – College-level benchmark comprising multiple-choice and open-ended questions from 30 disciplines. | accuracy | zs_cot | Yes +llm_judge | Text | LLM-as-a-Judge Preference Comparison – Uses a Nova Judge model to determine preference between paired responses (B compared with A) for your prompts, calculating the probability of B being preferred over A. | all | judge | No +humaneval | Text | HumanEval - A benchmark dataset designed to evaluate the code generation capabilities of large language models | pass@1 | zs | No + +The following `mmlu` subtasks are available: + + + MMLU_SUBTASKS = [ + "abstract_algebra", + "anatomy", + "astronomy", + "business_ethics", + "clinical_knowledge", + "college_biology", + "college_chemistry", + "college_computer_science", + "college_mathematics", + "college_medicine", + "college_physics", + "computer_security", + "conceptual_physics", + "econometrics", + "electrical_engineering", + "elementary_mathematics", + "formal_logic", + "global_facts", + "high_school_biology", + "high_school_chemistry", + "high_school_computer_science", + "high_school_european_history", + "high_school_geography", + "high_school_government_and_politics", + "high_school_macroeconomics", + "high_school_mathematics", + "high_school_microeconomics", + "high_school_physics", + "high_school_psychology", + "high_school_statistics", + "high_school_us_history", + "high_school_world_history", + "human_aging", + "human_sexuality", + "international_law", + "jurisprudence", + "logical_fallacies", + "machine_learning", + "management", + "marketing", + "medical_genetics", + "miscellaneous", + "moral_disputes", + "moral_scenarios", + "nutrition", + "philosophy", + "prehistory", + "professional_accounting", + "professional_law", + "professional_medicine", + "professional_psychology", + "public_relations", + "security_studies", + "sociology", + "us_foreign_policy", + "virology", + "world_religions" + ] + +The following `bbh` subtasks are available: + + + BBH_SUBTASKS = [ + "boolean_expressions", + "causal_judgement", + "date_understanding", + "disambiguation_qa", + "dyck_languages", + "formal_fallacies", + "geometric_shapes", + "hyperbaton", + "logical_deduction_five_objects", + "logical_deduction_seven_objects", + "logical_deduction_three_objects", + "movie_recommendation", + "multistep_arithmetic_two", + "navigate", + "object_counting", + "penguins_in_a_table", + "reasoning_about_colored_objects", + "ruin_names", + "salient_translation_error_detection", + "snarks", + "sports_understanding", + "temporal_sequences", + "tracking_shuffled_objects_five_objects", + "tracking_shuffled_objects_seven_objects", + "tracking_shuffled_objects_three_objects", + "web_of_lies", + "word_sorting" + ] + +The following `math` subtasks are available: + + + MATH_SUBTASKS = [ + "algebra", + "counting_and_probability", + "geometry", + "intermediate_algebra", + "number_theory", + "prealgebra", + "precalculus", + ] + +## Understanding the recipe parameters + +###### Run configuration + +The following is a general run configuration and an explanation of the parameters involved. + + + run: + name: eval_job_name + model_type: amazon.nova-micro-v1:0:128k + model_name_or_path: nova-micro/prod + replicas: 1 + data_s3_path: "" + output_s3_path: s3://output_path + + * `name`: (Required) A descriptive name for your evaluation job. This helps identify your job in the AWS console. + + * `model_type`: (Required) Specifies the Amazon Nova model variant to use. Do not manually modify this field. Options include: + + * `amazon.nova-micro-v1:0:128k` + + * `amazon.nova-lite-v1:0:300k` + + * `amazon.nova-pro-v1:0:300k` + + * `model_name_or_path`: (Required) The path to the base model or S3 path for the post-trained checkpoint. Options include: + + * `nova-micro/prod` + + * `nova-lite/prod` + + * `nova-pro/prod` + + * (S3 path for the post-trained checkpoint) `s3://<escrow bucket>/<job id>/outputs/checkpoints` + + * `replicas`: (Required) The number of compute instances to use for distributed training. You must set this value to 1 because multi-node is not supported. + + * `data_s3_path`: (Required) The S3 path to the input dataset. Leave this parameter empty unless you are using the _bring your own dataset_ or _LLM as a judge_ recipe. + + * `output_s3_path`: (Required) The S3 path to store output evaluation artifacts. Note that the output S3 bucket must be created by the same account that is creating the job. + + +