AWS sagemaker documentation change
Summary
Added limitations section for Bring Your Own Metrics regarding multi-modal input support
Security assessment
Documentation of feature limitations doesn't indicate security fixes or introduce security controls.
Diff
diff --git a/sagemaker/latest/dg/nova-model-evaluation.md b/sagemaker/latest/dg/nova-model-evaluation.md index dc22d493e..1b54f0643 100644 --- a//sagemaker/latest/dg/nova-model-evaluation.md +++ b//sagemaker/latest/dg/nova-model-evaluation.md @@ -62 +62 @@ ifeval | Text | Instruction-Following Evaluation – Gauges how accurately a mo -gen_qa | Text | Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. | all | gen_qa | No +gen_qa | Multi-Modal (image) | Custom Dataset Evaluation – Lets you supply your own dataset for benchmarking, comparing model outputs to reference answers with metrics such as ROUGE and BLEU. `gen_qa` supports image inference for Amazon Nova Lite or Amazon Nova Pro based models. | all | gen_qa | No @@ -64,0 +65 @@ llm_judge | Text | LLM-as-a-Judge Preference Comparison – Uses a Nova Judge m +mm_llm_judge | Multi-Modal (image) | This new benchmark behaves the same as the text-based `llm_judge`above. The only difference is that it supports image inference. | all | judge | No @@ -108 +109 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste - * `replica`: The number of compute instances to use for distributed training. Set to 1 as multi-node is not supported. + * `replica`: The number of compute instances to use for distributed inference (running inference across multiple nodes). Set `replica` > 1 to enable multi-node inference, which accelerates evaluation. If both `instance_count` and `replica` are specified, `instance_count` takes precedence. Note that multiple replicas only apply to SageMaker training jobs, not SageMaker HyperPod. @@ -145,0 +147,2 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste + * `mm_llm_judge` + @@ -156 +159 @@ To evaluate a post-trained model after a Nova SFT training job, follow these ste - * `judge`: Strategy specific for Nova LLM as Judge. + * `judge`: Strategy specific for Nova LLM as Judge and `mm_llm_judge`. @@ -192 +195 @@ For `gen_qa`, bring your own dataset benchmark, return following metrics: -For `llm_judge`, bring your own dataset benchmark, return following metrics: +For `llm_judge` and `mm_llm_judge`, bring your own dataset benchmark, return following metrics: @@ -230,0 +234 @@ For `llm_judge`, bring your own dataset benchmark, return following metrics: + top_logprobs: 10 @@ -239,0 +244,2 @@ For `llm_judge`, bring your own dataset benchmark, return following metrics: + * `top_logprobs`: The number of top logprobs to be returned in the inference response. This value must be an integer from 0 to 20. Logprobs contain the considered output tokens and log probabilities of each output token returned in the message content. + @@ -279 +285 @@ File format: - * The file must follow the required schema format for general Q&Q dataset. + * The file must follow the required schema format for general Q&A dataset. @@ -284 +290 @@ File format: -Schema format - Each line in the `.jsonl` file must be a JSON object with the following fields. +Schema format requirements - Each line in the `.jsonl` file must be a JSON object with the following fields. @@ -295,0 +302,4 @@ Schema format - Each line in the `.jsonl` file must be a JSON object with the fo +`images`: Array containing a list of objects with data attributes (Base64 encoded image strings). + +`metadata`: String containing metadata associated with the entry for tagging purposes. + @@ -313,0 +324,9 @@ Schema format - Each line in the `.jsonl` file must be a JSON object with the fo + }{ + "system": "Image inference: ", + "query": "What is the number in the image? Please just use one English word to answer.", + "response": "two", + "images": [ + { + "data": "data:image/png;Base64,iVBORw0KGgoA ..." + } + ] @@ -316 +335 @@ Schema format - Each line in the `.jsonl` file must be a JSON object with the fo -To use your custom dataset, modify your evaluation recipe with the following required fields, don't change any of the content: +To use your custom dataset, modify your evaluation recipe by adding the following required fields without changing the existing configuration: @@ -328 +347,179 @@ To use your custom dataset, modify your evaluation recipe with the following req - * File must strictly follow the defined schema. + * The file must strictly follow the defined schema. + + + + +##### Bring your own metrics + +You can bring your own metrics to fully customize your model evaluation workflow with custom preprocessing, postprocessing, and metrics capabilities. Preprocessing allows you to process input data before sending it to the inference server, and postprocessing allows you to customize metrics calculation and return custom metrics based on your needs. + +Follow these steps to bring your own metrics with custom evaluation SDK. + + 1. If you haven't done so, [create an AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) in your AWS account first. + + 2. Download the pre-built `nova-custom-eval-layer.zip` file from the [GitHub repository](https://github.com/aws/nova-custom-eval-sdk/releases). You can use this open-source Nova custom evaluation SDK to validate input and output payloads for your custom function and provide a unified interface for integrating with Nova's bring your own metrics evaluation during training. + + 3. Upload the custom Lambda layer using the following command: + + aws lambda publish-layer-version \ + --layer-name nova-custom-eval-layer \ + --zip-file fileb://nova-custom-eval-layer.zip \ + --compatible-runtimes python3.12 python3.11 python3.10 python3.9 + + 4. Add this layer as a custom layer to your Lambda function, along with the required AWS layer: `AWSLambdaPowertoolsPythonV3-python312-arm64` (required for `pydantic` dependency). + + 5. Update your Lambda code using the provided example, modifying the code as needed. This example code creates a Lambda function for Nova's custom evaluation with preprocessing and postprocessing steps for model evaluation. + + from nova_custom_evaluation_sdk.processors.decorators import preprocess, postprocess + from nova_custom_evaluation_sdk.lambda_handler import build_lambda_handler + + @preprocess + def preprocessor(event: dict, context) -> dict: + data = event.get('data', {}) + return { + "statusCode": 200, + "body": { + "system": data.get("system"), + "prompt": data.get("prompt", ""), + "gold": data.get("gold", "") + } + } + + @postprocess + def postprocessor(event: dict, context) -> dict: + # data is already validated and extracted from event + data = event.get('data', []) + inference_output = data.get('inference_output', '') + gold = data.get('gold', '') + + metrics = [] + inverted_accuracy = 0 if inference_output.lower() == gold.lower() else 1.0 + metrics.append({ + "metric": "inverted_accuracy_custom", + "value": accuracy + }) + + # Add more metrics here + + return { + "statusCode": 200, + "body": metrics + } + + # Build Lambda handler + lambda_handler = build_lambda_handler( + preprocessor=preprocessor, + postprocessor=postprocessor + ) + + 6. Grant Lambda access to the evaluation job. Ensure the execution role specified for the evaluation job includes a policy the invoke your Lambda function. Here is an example policy. + + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "LambdaAccess", + "Effect": "Allow", + "Action": [ + "lambda:InvokeFunction" + ], + "Resource": "arn:aws:lambda:us-east-1:111122223333:function:ExampleFunction", + "Condition": { + "StringLike": { + "aws:PrincipalArn": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole-ARN" + } + } + }, + { + "Sid": "DenyNonAWSEventSourcesForLambda", + "Effect": "Deny", + "Action": [ + "lambda:InvokeFunction" + ], + "Resource": "arn:aws:lambda:us-east-1:111122223333:function:ExampleFunction", + "Condition": { + "Null": { + "lambda:EventSourceToken": false + } + } + } + ] + } + + 7. Review the Lambda payload schema. The following table lists the Lambda request and response schema. You can validate your schema using the Nova custom evaluation SDK. + +| Lambda Request Payload | Lambda Response Payload +---|---|--- +Preprocessor | + + { + "process_type": "preprocess", + "data": { + "system": "You are a helpful assistant", + "prompt": "What is 2+2?", + "gold": "4" + } + } + +| + + { + "statusCode": 200, + "body": { + "system": "You are a helpful assistant that can substitute * for addition", + "prompt": "What is 2*2?", + "gold": "4" + } + } + +Postprocessor | + + { + "process_type": "postprocess", + "data": { + "prompt": "What is 2+2?", + "inference_output": "2+2=4", + "gold": "4" + } + } + +| + + { + "statusCode": 200, + "body": [ + {"metric": "accuracy", "value": 1.0}, + {"metric": "f1_score", "value": 1.0}, + {"metric": "exact_match", "value": 1}, + {"metric": "length_ratio", "value": 0.8}