AWS bedrock documentation change
Summary
Updated documentation for creating human-based model evaluation jobs with restructured steps, added AWS CLI examples, and clarified UI navigation paths
Security assessment
Changes focus on workflow clarification and example expansion. While KMS key and IAM role references exist, they document existing security practices rather than addressing vulnerabilities or introducing new security features
Diff
diff --git a/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.md b/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.md index b8408b475..30de09cce 100644 --- a/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.md +++ b/bedrock/latest/userguide/model-evaluation-jobs-management-create-human.md @@ -5 +5 @@ -# Human-based model evaluation jobs +# Create a human-based model evaluation job @@ -7 +7 @@ -The following examples show how to create a model evaluation job that uses human workers. In the API, you can also include an [inference profile](./inference-profiles.html) in the job by specifying its ARN in the `modelIdentifier` field. +The following examples show how to create a model evaluation job that uses human workers. @@ -13 +13 @@ The following examples show how to create a model evaluation job that uses human - 1. Open the Amazon Bedrock console: [https://console.aws.amazon.com/bedrock/](https://console.aws.amazon.com/bedrock/) + 1. Open the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/). @@ -15 +15 @@ The following examples show how to create a model evaluation job that uses human - 2. In the navigation pane, choose **Model evaluation**. + 2. In the navigation pane, under **Inference and Assessment** , select **Evaluations**. @@ -17 +17 @@ The following examples show how to create a model evaluation job that uses human - 3. In the **Build an evaluation** card, under **Human: bring your own team** choose **Create human-based evaluation**. + 3. In the **Model evaluation** pane, under **Human** , choose **Create** and select **Human: Bring your own work team**. @@ -25 +25 @@ The following examples show how to create a model evaluation job that uses human - 5. Then, choose **Next**. + 3. Choose **Next**. @@ -27 +27 @@ The following examples show how to create a model evaluation job that uses human - 6. On the **Set up evaluation** page provide the following. + 5. On the **Set up evaluation** page, under **Inference source** , select the source for your model evaluation. You can evaluate the performance of Amazon Bedrock models, or of other models by providing your own inference response data in your prompt dataset. You can select up to two inference sources. For jobs with two sources, you don't have to choose the same type for both sources; you can select one Amazon Bedrock model, and provide your own inference response data for the second source. To evaluate Amazon Bedrock models, do the following: @@ -29 +29 @@ The following examples show how to create a model evaluation job that uses human - 1. **Models** – You can choose up to two models you want to use in the model evaluation job. + 1. Under **Select source** , select **Bedrock models**. @@ -31 +31 @@ The following examples show how to create a model evaluation job that uses human -To learn more about available models in Amazon Bedrock, see [Access Amazon Bedrock foundation models](./model-access.html). + 2. Choose **Select model** to choose the model you want to evaluate. @@ -33 +33 @@ To learn more about available models in Amazon Bedrock, see [Access Amazon Bedro - 2. (Optional) To change the inference configuration for the selected models choose **update**. + 3. To select a second model, choose **Add model** and repeat the preceding steps. @@ -35 +35 @@ To learn more about available models in Amazon Bedrock, see [Access Amazon Bedro -Changing the inference configuration changes the responses generated by the selected models. To learn more about the available inferences parameters, see [Inference request parameters and response fields for foundation models](./model-parameters.html). + 6. To bring your own inference response data, do the following: @@ -37 +37 @@ Changing the inference configuration changes the responses generated by the sele - 3. **Task type** – Choose the type of task you want the model to attempt to perform during the model evaluation job. All instructions for the model must be included in the prompts themselves. The task type does not control the model's responses. + 1. Under **Select source** , select **Bring your own inference responses**. @@ -39 +39 @@ Changing the inference configuration changes the responses generated by the sele - 4. **Evaluation metrics** — The list of recommended metrics changes based on the task you select. For each recommended metric, you must select a **Rating method**. You can have a maximum of 10 evaluation metrics per model evaluation job. + 2. For **Source Name** , enter a name for the model you used to create the response data. The name you enter must match the `modelIdentifier` parameter in your [prompt dataset](./model-evaluation-prompt-datasets-custom-human.html#model-evaluation-prompt-datasets-custom-human-byoir). @@ -41 +41 @@ Changing the inference configuration changes the responses generated by the sele - 5. (Optional) Choose **Add metric** to add a metric. You must define the **Metric** , **Description** , and **Rating method**. + 3. To add a second source, choose **Add model** and repeat the preceding steps. @@ -43 +43 @@ Changing the inference configuration changes the responses generated by the sele - 6. In the **Datasets** card you must provide the following. + 7. For **Task type** , select the type of task you want the model to perform during the model evaluation job. All instructions for the model must be included in the prompts themselves. The task type does not control the model's responses. @@ -45 +45 @@ Changing the inference configuration changes the responses generated by the sele - 1. **Choose a prompt dataset** – Specify the S3 URI of your prompt dataset file or choose **Browse S3** to see available S3 buckets. You can have a maximum of 1000 prompts in a custom prompt dataset. + 8. In the **Datasets** pane, provide the following. @@ -47 +47 @@ Changing the inference configuration changes the responses generated by the sele - 2. **Evaluation results destination** – You must specify the S3 URI of the directory where you want the results of your model evaluation job saved, or choose **Browse S3** to see available S3 buckets. + 1. Under **Choose a prompt dataset** , specify the S3 URI of your prompt dataset file or choose **Browse S3** to see available S3 buckets. You can have a maximum of 1000 prompts in a custom prompt dataset. @@ -49 +49 @@ Changing the inference configuration changes the responses generated by the sele - 7. (Optional) **AWS KMS key** – Provide the ARN of the customer managed key you want to use to encrypt your model evaluation job. + 2. Under **Evaluation results destination** , specify the S3 URI of the directory where you want the results of your model evaluation job saved, or choose **Browse S3** to see available S3 buckets. @@ -51 +51,3 @@ Changing the inference configuration changes the responses generated by the sele - 8. In the **Amazon Bedrock IAM role – Permissions** card, you must do the following. To learn more about the required permissions for model evaluations, see [Service role requirements for model evaluation jobs](./model-evaluation-security-service-roles.html). + 9. (Optional) Under **KMS key - Optional** , provide the ARN of a customer managed key you want to use to encrypt your model evaluation job. + + 10. In the **Amazon Bedrock IAM role – Permissions** pane, do the following. To learn more about the required permissions for model evaluations, see [Service role requirements for model evaluation jobs](./model-evaluation-security-service-roles.html). @@ -59,7 +61 @@ Changing the inference configuration changes the responses generated by the sele - 7. Then, choose **Next**. - - 8. In the **Permissions** card, specify the following. To learn more about the required permissions for model evaluations, see [Service role requirements for model evaluation jobs](./model-evaluation-security-service-roles.html). - - 9. **Human workflow IAM role** – Specify a SageMaker AI service role that has the required permissions. - - 10. In the **Work team** card, specify the following. + 11. Choose **Next**. @@ -67 +63 @@ Changing the inference configuration changes the responses generated by the sele -###### Human worker notification requirements + 12. Under **Work team** , use the **Select team** dropdown to select an existing team, or create a new team by doing the following: @@ -69 +65 @@ Changing the inference configuration changes the responses generated by the sele -When you add a new human worker to a model evaluation job, they automatically receive an email inviting them to participate in the model evaluation job. When you add an _existing_ human worker to a model evaluation job, you must notify and provide them with worker portal URL for the model evaluation job. The existing worker will not receive an automated email notification that they are added to the new model evaluation job. + 1. Under **Team name** , enter a name for your team. @@ -71 +67 @@ When you add a new human worker to a model evaluation job, they automatically re - 1. Using the **Select team** dropdown, specify either **Create a new work team** or the name of an existing work team. + 2. Under **Email addresses** , enter the email addresses of the human workers in your team. @@ -73,5 +69 @@ When you add a new human worker to a model evaluation job, they automatically re - 2. (Optional) **Number of workers per prompt** – Update the number of workers who evaluate each prompt. After the responses for each prompt have been reviewed by the number of workers you selected, the prompt and its responses will be taken out of circulation from the work team. The final results report will include all ratings from each worker. - - 3. (Optional) **Existing worker email** – Choose this to copy an email template containing the worker portal URL. - - 4. (Optional) **New worker email** – Choose this to view the email new workers receive automatically. + 3. Under **Number of workers per prompt** , select the number of workers who evaluate each prompt. After the responses for each prompt have been reviewed by the number of workers you selected, the prompt and its responses will be taken out of circulation from the work team. The final results report will include all ratings from each worker. @@ -83 +75 @@ Large language models are known to occasionally hallucinate and produce toxic or - 11. Then, choose **Next**. + 13. Under **Human workflow IAM role - Permissions** , select an existing role, or select **Create a new role**. @@ -85 +77 @@ Large language models are known to occasionally hallucinate and produce toxic or - 12. On the **Provide instruction page** use the text editor to provide instructions for completing the task. You can preview the evaluation UI that your work team uses to evaluate the responses, including the metrics, rating methods, and your instructions. This preview is based on the configuration you have created for this job. + 14. Choose **Next**. @@ -87 +79 @@ Large language models are known to occasionally hallucinate and produce toxic or - 13. Then, choose **Next**. + 15. Under **Evaluation instructions** , provide instructions for completing the task. You can preview the evaluation UI that your work team uses to evaluate the responses, including the metrics, rating methods, and your instructions. This preview is based on the configuration you have created for this job. @@ -89 +81 @@ Large language models are known to occasionally hallucinate and produce toxic or - 14. On the **Review and create** page, you can view a summary of the options you've selected in the previous steps. + 16. Choose **Next**. @@ -91 +83 @@ Large language models are known to occasionally hallucinate and produce toxic or - 15. To start your model evaluation job, choose **Create**. + 17. Review your configuration and choose **Create** to create the job. @@ -133 +125 @@ The following is an example request using the AWS CLI. In the request, the `Huma - "S3OutputPath": "s3://your-output-bucket" + "S3OutputPath": "s3://amzn-s3-demo-destination-bucket" @@ -139,0 +132,68 @@ After creating your flow definition ARN, use the following examples to create hu +AWS CLI + + +The following example command and JSON file shows you how to create a model evaluation job using human workers where you provide your own inference response data. To learn how to specify a prompt dataset for a model evaluation job with human workers, see [Create a custom prompt dataset for a model evaluation job that uses human workers](./model-evaluation-prompt-datasets-custom-human.html). + +###### Example AWS CLI command and JSON file to create an evaluation job using your own inference response data + + + aws bedrock create-evaluation-job --cli-input-json file://my_eval_job.json + + + { + "jobName": "model-eval-llama-vs-my-other-model", + "roleArn": "arn:aws:iam::111122223333:role/service-role/Amazon-Bedrock-IAM-Role-20250218T223671", + "evaluationConfig": { + "human": { + "customMetrics": [ + { + "description": "Measures the organization and structure of a generated text.", + "name": "Coherence", + "ratingMethod": "ThumbsUpDown" + }, + { + "description": "Indicates the accuracy of a generated text.", + "name": "Accuracy", + "ratingMethod": "ComparisonChoice" + } + ], + "datasetMetricConfigs": [ + { + "dataset": { + "datasetLocation": { + "s3Uri": "s3://amzn-s3-demo-bucket/input/model-eval/fitness-dataset-model-eval-byoir-2-models.jsonl" + }, + "name": "dataset1" + }, + "metricNames": [ + "Coherence", + "Accuracy" + ], + "taskType": "Generation" + } + ], + "humanWorkflowConfig": { + "flowDefinitionArn": "arn:aws:sagemaker:us-east-1:111122223333:flow-definition/bedrock-fitness-human-byoir", + "instructions": "<h3>The following are the metrics and their descriptions for this evaluation</h3>\n<p><strong>Coherence</strong>: Measures the organization and structure of a generated text. - <em>Thumbs up/down</em>\n<strong>Accuracy</strong>: Indicates the accuracy of a generated text. - <em>Choice buttons</em></p>\n<h3>Instructions for how to use the evaluation tool</h3>\n<p>The evaluation creator should use this space to write detailed descriptions for every rating method so your evaluators know how to properly rate the responses with the buttons on their screen.</p>\n<h4>For example:</h4>\n<p>If using <strong>Likert scale - individual</strong>, define the 1 and 5 of the 5 point Likert scale for each metric so your evaluators know if 1 or 5 means favorable/acceptable/preferable.\nIf using <strong>Likert scale - comparison</strong>, describe what the evaluator is looking for to determine their preference between two responses.\nIf using <strong>Choice buttons</strong>, describe what is preferred according to your metric and its description.\nIf using <strong>Ordinal ranking</strong>, define what should receive a #1 ranking according to your metric and its description.\nIf using <strong>Thumbs up/down</strong>, define what makes an acceptable response according to your metric and its description.</p>\n<h3>Describing your ground truth responses if applicable to your dataset</h3>\n<p>Describe the purpose of your ground truth responses that will be shown on screen next to each model response. Note that the ground truth responses you provide are not rated/scored by the evaluators - they are meant to be a reference standard for comparison against the model responses.</p>" + } + } + }, + "inferenceConfig": { + "models": [ + { + "precomputedInferenceSource": { + "inferenceSourceIdentifier": "llama-3-1-80b" + } + }, + { + "precomputedInferenceSource": { + "inferenceSourceIdentifier": "my_other_model" + } + } + ] + }, + "outputDataConfig": { + "s3Uri": "s3://amzn-s3-demo-bucket/output/" + } + } +