AWS bedrock documentation change
Summary
Updated documentation for creating model evaluation jobs, including clarifications on model types, updated AWS CLI examples with detailed configurations, and replacing SDK examples with CLI commands.
Security assessment
The changes involve documentation updates, example corrections, and configuration details without any mention of security vulnerabilities, patches, or security-specific features. Modifications focus on workflow clarity, parameter specifications, and example standardization.
Diff
diff --git a/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.md b/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.md index 50b48257e..48dfccc41 100644 --- a/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.md +++ b/bedrock/latest/userguide/model-evaluation-jobs-management-create-judge.md @@ -14,0 +15,2 @@ This job requires two different models. Both models must be available in the sam + * Amazon Bedrock Marketplace models – [Amazon Bedrock Marketplace](./amazon-bedrock-marketplace.html) + @@ -17 +19,3 @@ This job requires two different models. Both models must be available in the sam - * Imported models – [Import a customized model into Amazon Bedrock](./model-customization-import-model.html) + * Imported foundation models – [Import a customized model into Amazon Bedrock](./model-customization-import-model.html) + + * Prompt routers – [Understanding intelligent prompt routing in Amazon Bedrock](./prompt-routing.html) @@ -19 +23 @@ This job requires two different models. Both models must be available in the sam - * Models with Provisioned Throughput – [Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock](./prov-throughput.html) + * Models for which you have purchased Provisioned Throughput – [Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock](./prov-throughput.html) @@ -39 +43 @@ This job requires two different models. Both models must be available in the sam -Use the following tabs to learn how to make a [`CreateEvaluation`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateEvaluationJob.html). Use `inferenceConfig` to specify the generator model you want to use in the model evaluation job. Use `evaluatorModelConfig` to specify the supported evaluator model. +Use the following to learn how to create an evaluation job using the [`CreateEvaluationJob`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateEvaluationJob.html) API. Use `inferenceConfig` to specify the generator model you want to use in the model evaluation job. Use `evaluatorModelConfig` to specify the supported evaluator model. @@ -44 +48,5 @@ AWS CLI -The following example show how to make a `create-evaluation-job` request using the AWS CLI. Use the table found in _model metrics for model as judge_ to see how to correctly format the `metricNames`. Ensure that `evaluationContext` is specified as `Model`. +The following examples show how to make a `create-evaluation-job` request using the AWS CLI. Ensure that `applicationType` is specified as `ModelEvaluation`. + +You can either evaluate the performance of models in Amazon Bedrock, or you can evaluate other models by providing your own inference response data as part of the prompt dataset. To learn more about how to create a prompt dataset using your own inference responses, see [Create a custom prompt dataset for a model evaluation job that uses a model as judge](./model-evaluation-prompt-datasets-judge.html). + +###### Example AWS CLI command and JSON file to create an evaluation job for an Amazon Bedrock model @@ -47 +55 @@ The following example show how to make a `create-evaluation-job` request using t - aws bedrock create-evaluation-job + aws bedrock create-evaluation-job --cli-input-json file://my_eval_job.json @@ -53 +61 @@ The following example show how to make a `create-evaluation-job` request using t - "evaluationContext": "Model", + "applicationType": "ModelEvaluation", @@ -62 +70 @@ The following example show how to make a `create-evaluation-job` request using t - "s3Uri": "s3://bedrock-model-as-a-judge-test-1/input_datasets/text_dataset_input.jsonl" + "s3Uri": "s3://amzn-s3-demo-bucket/input_datasets/text_dataset_input.jsonl" @@ -85 +93 @@ The following example show how to make a `create-evaluation-job` request using t - "inferenceParams": "{}" + "inferenceParams": "{\"inferenceConfig\":{\"maxTokens\":512,\"temperature\":1,\"topP\":0.999,\"stopSequences\":[\"stop\"]},\"additionalModelRequestFields\":{\"top_k\": 128}}" @@ -91 +99 @@ The following example show how to make a `create-evaluation-job` request using t - "s3Uri": "s3://bedrock-model-as-a-judge-test-1/output_data/" + "s3Uri": "s3://amzn-s3-demo-bucket/output_data/" @@ -95,8 +103 @@ The following example show how to make a `create-evaluation-job` request using t -SDK for Python - - -When you create a human-based model evaluation job outside of the Amazon Bedrock console, you need to create an Amazon SageMaker AI flow definition ARN. - -The flow definition ARN is where a model evaluation job's workflow is defined. The flow definition is used to define the worker interface and the work team you want assigned to the task, and connecting to Amazon Bedrock. - -For model evaluation jobs started using Amazon Bedrock API operations you _must_ create a flow definition ARN using the AWS CLI or a supported AWS SDK. To learn more about how flow definitions work, and creating them programmatically, see [Create a Human Review Workflow (API)](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html#a2i-create-human-review-api) in the _SageMaker AI Developer Guide_. +###### Example AWS CLI command and JSON file to create an evaluation job where you provide your own inference response data @@ -104 +104,0 @@ For model evaluation jobs started using Amazon Bedrock API operations you _must_ -In the [`CreateFlowDefinition`](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html) you must specify `AWS/Bedrock/Evaluation` as input to the `AwsManagedHumanLoopRequestSource`. The Amazon Bedrock service role must also have permissions to access the output bucket of the flow definition. @@ -106 +106 @@ In the [`CreateFlowDefinition`](https://docs.aws.amazon.com/sagemaker/latest/API -The following is an example request using the AWS CLI. In the request, the `HumanTaskUiArn` is a SageMaker AI owned ARN. In the ARN, you can only modify the AWS Region. + aws bedrock create-evaluation-job --cli-input-json file://my_eval_job.json @@ -109 +108,0 @@ The following is an example request using the AWS CLI. In the request, the `Huma - aws sagemaker create-flow-definition --cli-input-json ' @@ -111,36 +110,5 @@ The following is an example request using the AWS CLI. In the request, the `Huma - "FlowDefinitionName": "human-evaluation-task01", - "HumanLoopRequestSource": { - "AwsManagedHumanLoopRequestSource": "AWS/Bedrock/Evaluation" - }, - "HumanLoopConfig": { - "WorkteamArn": "arn:aws:sagemaker:AWS Region:111122223333:workteam/private-crowd/my-workteam", - ## The Task UI ARN is provided by the service team, you can only modify the AWS Region. - "HumanTaskUiArn":"arn:aws:sagemaker:AWS Region:394669845002:human-task-ui/Evaluation" - "TaskTitle": "Human review tasks", - "TaskDescription": "Provide a real good answer", - "TaskCount": 1, - "TaskAvailabilityLifetimeInSeconds": 864000, - "TaskTimeLimitInSeconds": 3600, - "TaskKeywords": [ - "foo" - ] - }, - "OutputConfig": { - "S3OutputPath": "s3://your-output-bucket" - }, - "RoleArn": "arn:aws:iam::111122223333:role/SageMakerCustomerRoleArn" - }' - -The following code example shows you how to create a model evaluation job that uses human workers via the SDK for SDK for Python. - - - import boto3 - client = boto3.client('bedrock') - - job_request = client.create_evaluation_job( - jobName="111122223333-job-01", - jobDescription="two different task types", - roleArn="arn:aws:iam::111122223333:role/example-human-eval-api-role", - inferenceConfig={ - ## You must specify and array of models - "models": [ + "jobName": "model-eval-llmaj", + "roleArn": "arn:aws:iam::111122223333:role/Amazon-Bedrock-ModelAsAJudgeTest", + "evaluationConfig": { + "automated": { + "datasetMetricConfigs": [ @@ -148,3 +116,5 @@ The following code example shows you how to create a model evaluation job that u - "bedrockModel": { - "modelIdentifier":"arn:aws:bedrock:us-west-2::foundation-model/amazon.titan-text-lite-v1", - "inferenceParams":"{\"temperature\":\"0.0\", \"topP\":\"1\", \"maxTokenCount\":\"512\"}" + "taskType": "General", + "dataset": { + "name": "text_dataset", + "datasetLocation": { + "s3Uri": "s3://amzn-s3-demo-bucket/input/model-eval/fitness-dataset-model-eval-byoi.jsonl" @@ -154,6 +123,3 @@ The following code example shows you how to create a model evaluation job that u - { - "bedrockModel": { - "modelIdentifier": "anthropic.claude-v2", - "inferenceParams": "{\"temperature\":\"0.25\",\"top_p\":\"0.25\",\"max_tokens_to_sample\":\"256\",\"top_k\":\"1\"}" - } - } + "metricNames": [ + "Builtin.Correctness", + "Builtin.Completeness" @@ -161,16 +126,0 @@ The following code example shows you how to create a model evaluation job that u - - }, - outputDataConfig={ - "s3Uri":"s3://job-bucket/outputs/" - }, - evaluationConfig={ - "human": { - "humanWorkflowConfig": { - "flowDefinitionArn": "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/example-workflow-arn", - "instructions": "some human eval instruction" - }, - "customMetrics": [ - { - "name": "IndividualLikertScale", - "description": "testing", - "ratingMethod": "IndividualLikertScale" @@ -179 +129,2 @@ The following code example shows you how to create a model evaluation job that u - "datasetMetricConfigs": [ + "evaluatorModelConfig": { + "bedrockEvaluatorModels": [ @@ -181,5 +132 @@ The following code example shows you how to create a model evaluation job that u - "taskType": "Summarization", - "dataset": { - "name": "Custom_Dataset1", - "datasetLocation": { - "s3Uri": "s3://job-bucket/custom-datasets/custom-trex.jsonl" + "modelIdentifier": "us.meta.llama3-1-70b-instruct-v1:0" @@ -187,3 +133,0 @@ The following code example shows you how to create a model evaluation job that u - }, - "metricNames": [ - "IndividualLikertScale" @@ -191,0 +136,9 @@ The following code example shows you how to create a model evaluation job that u + } + }, + "inferenceConfig": { + "models": [ + { + "precomputedInferenceSource": { + "inferenceSourceIdentifier": "my_model" + } + } @@ -192,0 +146,3 @@ The following code example shows you how to create a model evaluation job that u + }, + "outputDataConfig": { + "s3Uri": "s3://amzn-s3-demo-bucket/output/" @@ -196,3 +150,0 @@ The following code example shows you how to create a model evaluation job that u - ) - - print(job_request)