AWS bedrock documentation change
Summary
Restructured content to focus on prerequisites, IAM policies, and removed CLI examples in favor of topic links. Added details about required service roles, CORS permissions, and example IAM policies.
Security assessment
The changes document security-related configurations like IAM service role requirements and S3 bucket CORS permissions, which are security best practices. However, there is no evidence these changes address a specific disclosed vulnerability or security incident.
Diff
diff --git a/bedrock/latest/userguide/knowledge-base-evaluation-create.md b/bedrock/latest/userguide/knowledge-base-evaluation-create.md index 06fabc3fb..86b959ade 100644 --- a//bedrock/latest/userguide/knowledge-base-evaluation-create.md +++ b//bedrock/latest/userguide/knowledge-base-evaluation-create.md @@ -5 +5 @@ -RAG evaluation jobs that use LLMsRAG evaluation jobs using your own inference response data +Prerequisites @@ -9 +9 @@ RAG evaluation jobs that use LLMsRAG evaluation jobs using your own inference re -You can create a Retrieval Augmented Generation (RAG) evaluation job that computes metrics for the evaluation. If you want to evaluate the performance of your own non-Amazon Bedrock RAG source, you can do this by providing your own inference response data in the prompt dataset. +You can create a RAG evaluation job using the AWS Management Console, AWS CLI, or a supported AWS SDK. @@ -11 +11 @@ You can create a Retrieval Augmented Generation (RAG) evaluation job that comput -Refer to [Create a prompt dataset for a RAG evaluation in Amazon Bedrock](./knowledge-base-evaluation-prompt.html) to learn how to specify a prompt dataset for a knowledge base evaluation job. +This type of job requires access to an evaluator model. If you are creating a retrieve-and-generate job that uses an Amazon Bedrock model as the response generator, you need access to that model as well. Both models must be available in the same AWS Region. For a list of supported response generator and evaluator models, see [Supported models](./evaluation-kb.html#evaluation-kb-supported). To learn more about accessing models, see [Access Amazon Bedrock foundation models](./model-access.html). @@ -13 +13 @@ Refer to [Create a prompt dataset for a RAG evaluation in Amazon Bedrock](./know -Certain access permissions are required to create knowledge base evaluation jobs. For more information, see [Required permissions to create a Amazon Bedrock Knowledge Bases evaluation job](./knowledge-base-evaluation-prereq.html#knowledge-base-evaluation-prereq-create). +## Prerequisites @@ -15 +15 @@ Certain access permissions are required to create knowledge base evaluation jobs -You can evaluate retrieval only of your knowledge base or retrieval with response generation. Different metrics are relevant to retrieval only and retrieval with response generation. For more information, see [Review metrics for RAG evaluations that use LLMs (console)](./knowledge-base-eval-llm-results.html) +In addition to having access to at least one evaluator model, to create a RAG evaluation job, you also need certain IAM service role permissions. To learn more about the necessary actions and trust policy requirements, see [Required service role permissions for creating a model evaluation job that uses a judge model](./judge-service-roles.html). @@ -17 +17 @@ You can evaluate retrieval only of your knowledge base or retrieval with respons -You must choose a supported evaluator model to compute the metrics for your evaluation. If you want to evaluate retrieval with response generation for an Amazon Bedrock knowledge base, then you must also choose a supported model for response generation. For more information, see [Prerequisites for creating RAG evaluations in Amazon Bedrock](./knowledge-base-evaluation-prereq.html) +When you create the job, you specify a prompt dataset in an Amazon S3 bucket, and an output bucket to store your results in. To ensure your S3 buckets have the necessary CORS permissions, see [Required Cross Origin Resource Sharing (CORS) permissions on S3 buckets](./model-evaluation-security-cors.html) @@ -19 +19 @@ You must choose a supported evaluator model to compute the metrics for your eval -The following example shows you how to create a knowledge base evaluation job using the AWS CLI. +To create a job in the console, the console needs permission to perform a certain set of actions and have access to the needed resources. The following policy defines a minimum set of IAM permissions required to create a job in the console. In the policy, we recommend using the IAM JSON policy element [Resource](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html) to limit access to only the models and buckets required for the IAM user, group, or role. @@ -21 +21 @@ The following example shows you how to create a knowledge base evaluation job us -## RAG evaluation jobs that use LLMs +The IAM policy must grant access to both an evaluator model and, for retrieve and generate jobs that use an Amazon Bedrock response generator model, to the response generator. @@ -23,15 +22,0 @@ The following example shows you how to create a knowledge base evaluation job us -The following example shows you how to create a RAG evaluation job that uses Large Language Models (LLMs) for the evaluation. - -_AWS Command Line Interface_ - - - aws bedrock create-evaluation-job \ - --job-name "rag-evaluation-complete-stereotype-docs-app" \ - --job-description "Evaluates Completeness and Stereotyping of RAG for docs application" \ - --role-arn "arn:aws::iam:<region>:<account-id>:role/AmazonBedrock-KnowledgeBases" \ - --application-type "RagEvaluation" \ - --evaluationConfig file://knowledge-base-evaluation-config.json \ - --inference-config file://knowledge-base-evaluation-inference-config.json \ - --output-data-config '{"s3Uri":"s3://docs/kbevalresults/"}' - - file://knowledge-base-evaluation-config.json @@ -40,19 +25,2 @@ _AWS Command Line Interface_ - "automated": [{ - "datasetMetricConfigs": [{ - "taskType":"Generation", //Required field for model evaluation, but ignored/not used for knowledge base evaluation - "metricNames":["Builtin.Completeness","Builtin.Stereotyping"], - "dataset": [{ - "name":"RagTestPrompts", - "datasetLocation":"s3://docs/kbtestprompts.jsonl" - }] - }], - "evaluatorModelConfig": { - "bedrockEvaluatorModels": [{ - "modelIdentifier": "anthropic.claude-3-5-sonnet-20240620-v1:0" - }] - } - }] - } - - file://knowledge-base-evaluation-inference-config.json - + "Version": "2012-10-17", + "Statement": [ @@ -60,60 +28,21 @@ _AWS Command Line Interface_ - "ragConfigs": { - "knowledgeBaseConfig": [{ - "retrieveConfig": [{ - "knowledgeBaseId":"<knowledge-base-id>", - "knowledgeBaseRetrievalConfiguration": { - "vectorSearchConfiguration": [{ - "numberOfResults": 10, - "overrideSearchType":"HYBRID" - }] - } - }], - "retrieveAndGenerateConfig": [{ - "type":"KNOWLEDGE_BASE", - "knowledgeBaseConfiguration": [{ - "knowledgeBaseId":"<knowledge-base-id>", - "modelArn":"arn:aws:bedrock:<region>:<account-id>:inference-profile/anthropic.claude-v2:1", - "generationConfiguration": { - "promptTemplate": { - "textPromptTemplate": "\n\nHuman: I will provide you with a set of search results and a user's question. Your job is to answer the user's question using only information from the search results\n\nHere are the search results: $search_results$\n\nHere is the user's question: $query$\n\nAssistant:" - } - - } - }] - }] - }] - } - } - -_AWS SDK for Python (Boto3)_ - -###### Note - -During preview, your AWS account management will provide you with a parameters file to download and use. - -The following python example demonstrates how to make a retrieve-only Boto3 API request. To learn more about creating an evaluation job using Boto3, refer to [`create_evaluation_job`](https://boto3.amazonaws.com/v1/documentation/api/1.35.6/reference/services/bedrock/client/create_evaluation_job.html) in the Boto3 documentation. - - - import boto3 - client = boto3.client('bedrock') - - job_response = client.create_evaluation_job( - jobName="my_evaluation_job", - jobDescription="two different task types", - roleArn="arn:aws:iam::111122223333:role/service-role/Amazon-Bedrock-IAM-RoleAmazon-Bedrock-IAM-Role", - applicationType="RagEvaluation", - inferenceConfig={ - "ragConfigs": [ - { - "knowledgeBaseConfig": { - "retrieveConfig": { - "knowledgeBaseId": "your-knowledge-base-id", - "knowledgeBaseRetrievalConfiguration": { - "vectorSearchConfiguration": { - "numberOfResults": 10, - "overrideSearchType": "HYBRID" - } - } - } - } - } + "Sid": "BedrockConsole", + "Effect": "Allow", + "Action": [ + "bedrock:CreateEvaluationJob", + "bedrock:GetEvaluationJob", + "bedrock:ListEvaluationJobs", + "bedrock:StopEvaluationJob", + "bedrock:GetCustomModel", + "bedrock:ListCustomModels", + "bedrock:CreateProvisionedModelThroughput", + "bedrock:UpdateProvisionedModelThroughput", + "bedrock:GetProvisionedModelThroughput", + "bedrock:ListProvisionedModelThroughputs", + "bedrock:GetImportedModel", + "bedrock:ListImportedModels", + "bedrock:ListTagsForResource", + "bedrock:UntagResource", + "bedrock:TagResource" + ], + "Resource": [ + "arn:aws:bedrock:us-west-2::foundation-model/*;", @@ -122,6 +50,0 @@ The following python example demonstrates how to make a retrieve-only Boto3 API - outputDataConfig={ - "s3Uri":"s3://amzn-s3-demo-bucket-model-evaluations/outputs/" - }, - evaluationConfig={ - "automated": { - "datasetMetricConfigs": [ @@ -129,11 +52,8 @@ The following python example demonstrates how to make a retrieve-only Boto3 API - "taskType": "Summarization", - "dataset": { - "name": "RagDataset", - "datasetLocation": { - "s3Uri": "s3://amzn-s3-demo-bucket/input_data/data_3_rng.jsonl" - } - }, - "metricNames": [ - "Builtin.ContextCoverage" - ] - } + "Sid": "AllowConsoleS3AccessForModelEvaluation", + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:GetBucketCORS", + "s3:ListBucket", + "s3:ListBucketVersions", + "s3:GetBucketLocation" @@ -141,14 +61,3 @@ The following python example demonstrates how to make a retrieve-only Boto3 API - "evaluatorModelConfig": - { - "bedrockEvaluatorModels": [{ - "modelIdentifier": "meta.llama3-1-70b-instruct-v1:0" - }] - } - } - } - ) - - print(job_request) - -The following python example demonstrates how to make a _Retrieve and generate_ boto3 API request. - + "Resource": [ + "arn:aws:s3:::amzn-s3-demo-destination-bucket/*", + "arn:aws:s3:::input_datasets/prompts.jsonl" @@ -156,32 +64,0 @@ The following python example demonstrates how to make a _Retrieve and generate_ - import boto3 - client = boto3.client('bedrock') - - job_response = client.create_evaluation_job( - jobName="my_evaluation_job", - jobDescription="two different task types",