AWS nova documentation change
Summary
Removed detailed model evaluation documentation and replaced with link to SageMaker user guide section
Security assessment
The changes remove technical implementation details about model evaluation benchmarks and job configuration without any security-related content. This appears to be documentation simplification/streamlining rather than addressing security issues or adding security features.
Diff
diff --git a/nova/latest/userguide/customize-fine-tune-evaluate.md b/nova/latest/userguide/customize-fine-tune-evaluate.md index 513359627..1cef1950c 100644 --- a//nova/latest/userguide/customize-fine-tune-evaluate.md +++ b//nova/latest/userguide/customize-fine-tune-evaluate.md @@ -5,2 +4,0 @@ -Available benchmark tasksUnderstanding the recipe parametersEvaluation recipe examplesStart an evaluation jobAccess and visualize your results - @@ -13,575 +11 @@ Model evaluation is an offline process, where models are tested against fixed be -###### Topics - - * Available benchmark tasks - - * Understanding the recipe parameters - - * Evaluation recipe examples - - * Start an evaluation job - - * Access and visualize your results - - - - -## Available benchmark tasks - -A sample code package is available that demonstrates how to calculate benchmark metrics using the SageMaker AI model evaluation feature for Amazon Nova. To access the code packages, see [sample-Nova-lighteval-custom-task](https://github.com/aws-samples/sample-Nova-lighteval-custom-task/). - -Here is a list of the supported, available industry standard benchmarks. You can specify the following benchmarks in the `eval_task` parameter: - -Benchmark | Modality | Description | Metrics | Strategy | Subtask Available ----|---|---|---|---|--- -mmlu | Text | Multi-task Language Understanding – Tests knowledge across 57 subjects. | accuracy | zs_cot | Yes -mmlu_pro | Text | MMLU – Professional Subset – Focuses on professional domains such as law, medicine, accounting, and engineering. | accuracy | zs_cot | No -bbh | Text | Advanced Reasoning Tasks – A collection of challenging problems that test higher-level cognitive and problem-solving skills. | accuracy | zs_cot | Yes -gpqa | Text | General Physics Question Answering – Assesses comprehension of physics concepts and related problem-solving abilities. | accuracy | zs_cot | No -math | Text | Mathematical Problem Solving – Measures mathematical reasoning across topics including algebra, calculus, and word problems. | exact_match | zs_cot | Yes -strong_reject | Text | Quality-Control Task – Tests the model’s ability to detect and reject inappropriate, harmful, or incorrect content. | deflection | zs | Yes -IFEval | Text | Instruction-Following Evaluation – Gauges how accurately a model follows given instructions and completes tasks to specification. | accuracy | zs | No -gen_qa | Text | Custom Dataset Evaluation – Lets you bring your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. | all | gen_qa | No -mmmu | Multi-modal | Massive Multidiscipline Multimodal Understanding (MMMU) – College-level benchmark comprising multiple-choice and open-ended questions from 30 disciplines. | accuracy | zs_cot | Yes -llm_judge | Text | LLM-as-a-Judge Preference Comparison – Uses a Nova Judge model to determine preference between paired responses (B vs A) for your prompts, calculating the probability of B being preferred over A. | all | judge | No -humaneval | Text | HumanEval - A benchmark dataset designed to evaluate the code generation capabilities of large language models | pass@1 | zs | No - -The following `mmlu` subtasks are available: - - - MMLU_SUBTASKS = [ - "abstract_algebra", - "anatomy", - "astronomy", - "business_ethics", - "clinical_knowledge", - "college_biology", - "college_chemistry", - "college_computer_science", - "college_mathematics", - "college_medicine", - "college_physics", - "computer_security", - "conceptual_physics", - "econometrics", - "electrical_engineering", - "elementary_mathematics", - "formal_logic", - "global_facts", - "high_school_biology", - "high_school_chemistry", - "high_school_computer_science", - "high_school_european_history", - "high_school_geography", - "high_school_government_and_politics", - "high_school_macroeconomics", - "high_school_mathematics", - "high_school_microeconomics", - "high_school_physics", - "high_school_psychology", - "high_school_statistics", - "high_school_us_history", - "high_school_world_history", - "human_aging", - "human_sexuality", - "international_law", - "jurisprudence", - "logical_fallacies", - "machine_learning", - "management", - "marketing", - "medical_genetics", - "miscellaneous", - "moral_disputes", - "moral_scenarios", - "nutrition", - "philosophy", - "prehistory", - "professional_accounting", - "professional_law", - "professional_medicine", - "professional_psychology", - "public_relations", - "security_studies", - "sociology", - "us_foreign_policy", - "virology", - "world_religions" - ] - -The following `bbh` subtasks are available: - - - BBH_SUBTASKS = [ - "boolean_expressions", - "causal_judgement", - "date_understanding", - "disambiguation_qa", - "dyck_languages", - "formal_fallacies", - "geometric_shapes", - "hyperbaton", - "logical_deduction_five_objects", - "logical_deduction_seven_objects", - "logical_deduction_three_objects", - "movie_recommendation", - "multistep_arithmetic_two", - "navigate", - "object_counting", - "penguins_in_a_table", - "reasoning_about_colored_objects", - "ruin_names", - "salient_translation_error_detection", - "snarks", - "sports_understanding", - "temporal_sequences", - "tracking_shuffled_objects_five_objects", - "tracking_shuffled_objects_seven_objects", - "tracking_shuffled_objects_three_objects", - "web_of_lies", - "word_sorting" - ] - -The following `math` subtasks are available: - - - MATH_SUBTASKS = [ - "algebra", - "counting_and_probability", - "geometry", - "intermediate_algebra", - "number_theory", - "prealgebra", - "precalculus", - ] - -## Understanding the recipe parameters - -###### Run configuration - -The following is a general run configuration and an explanation of the parameters involved. - - - run: - name: eval_job_name - model_type: amazon.nova-micro-v1:0:128k - model_name_or_path: nova-micro/prod - replicas: 1 - data_s3_path: "" - output_s3_path: s3://output_path - - * `name`: (Required) A descriptive name for your evaluation job. This helps identify your job in the AWS console. - - * `model_type`: (Required) Specifies the Amazon Nova model variant to use. Do not manually modify this field. Options include: - - * `amazon.nova-micro-v1:0:128k` - - * `amazon.nova-lite-v1:0:300k` - - * `amazon.nova-pro-v1:0:300k` - - * `model_name_or_path`: (Required) The path to the base model or S3 path for the post-trained checkpoint. Options include: - - * `nova-micro/prod` - - * `nova-lite/prod` - - * `nova-pro/prod` - - * (S3 path for the post-trained checkpoint) `s3://<escrow bucket>/<job id>/outputs/checkpoints` - - * `replicas`: (Required) The number of compute instances to use for distributed training. You must set this value to 1 because multi-node is not supported. - - * `data_s3_path`: (Required) The S3 path to the input dataset. Leave this parameter empty unless you are using the _bring your own dataset_ or _LLM as a judge_ recipe. - - * `output_s3_path`: (Required) The S3 path to store output evaluation artifacts. Note that the output S3 bucket must be created by the same account that is creating the job. - - - - -###### Evaluation configuration - -The following is a model evaluation configuration and an explanation of the parameters involved. -