AWS bedrock documentation change
Summary
Updated documentation to rename 'knowledge base evaluation' to 'RAG evaluation', added support for custom inference response data, and expanded configuration examples
Security assessment
Changes focus on rebranding terminology (knowledge base → RAG) and adding functionality for custom inference data integration. No security vulnerabilities or specific security features are mentioned in the diff.
Diff
diff --git a/bedrock/latest/userguide/knowledge-base-evaluation-create.md b/bedrock/latest/userguide/knowledge-base-evaluation-create.md index c8aa05f8c..06fabc3fb 100644 --- a/bedrock/latest/userguide/knowledge-base-evaluation-create.md +++ b/bedrock/latest/userguide/knowledge-base-evaluation-create.md @@ -5 +5 @@ -Knowledge base evaluation jobs that use LLMs +RAG evaluation jobs that use LLMsRAG evaluation jobs using your own inference response data @@ -7 +7 @@ Knowledge base evaluation jobs that use LLMs -# Creating a knowledge base evaluation job in Amazon Bedrock +# Creating a RAG evaluation job in Amazon Bedrock @@ -9 +9 @@ Knowledge base evaluation jobs that use LLMs -You can create a knowledge base evaluation job that computes metrics for the evaluation. +You can create a Retrieval Augmented Generation (RAG) evaluation job that computes metrics for the evaluation. If you want to evaluate the performance of your own non-Amazon Bedrock RAG source, you can do this by providing your own inference response data in the prompt dataset. @@ -11,3 +11 @@ You can create a knowledge base evaluation job that computes metrics for the eva -Certain access permissions are required to create knowledge base evaluation jobs. For more information, see [Required permissions to create a Amazon Bedrock Knowledge Bases evaluation job](./knowledge-base-evaluation-prereq.html#knowledge-base-evaluation-prereq-create). - -###### Note +Refer to [Create a prompt dataset for a RAG evaluation in Amazon Bedrock](./knowledge-base-evaluation-prompt.html) to learn how to specify a prompt dataset for a knowledge base evaluation job. @@ -15,3 +13 @@ Certain access permissions are required to create knowledge base evaluation jobs -Knowledge base evaluation jobs are in preview mode and are subject to change. - -You can evaluate retrieval only of your knowledge base or retrieval with response generation. Different metrics are relevant to retrieval only and retrieval with response generation. For more information, see [Review metrics for knowledge base evaluations that use LLMs (console)](./knowledge-base-eval-llm-results.html) +Certain access permissions are required to create knowledge base evaluation jobs. For more information, see [Required permissions to create a Amazon Bedrock Knowledge Bases evaluation job](./knowledge-base-evaluation-prereq.html#knowledge-base-evaluation-prereq-create). @@ -19 +15 @@ You can evaluate retrieval only of your knowledge base or retrieval with respons -You must choose a supported evaluator model to compute the metrics for your evaluation. If you want to evaluate retrieval with response generation, then you must also choose a supported model for response generation. For more information, see [Prerequisites for creating knowledge base evaluations in Amazon Bedrock](./knowledge-base-evaluation-prereq.html) +You can evaluate retrieval only of your knowledge base or retrieval with response generation. Different metrics are relevant to retrieval only and retrieval with response generation. For more information, see [Review metrics for RAG evaluations that use LLMs (console)](./knowledge-base-eval-llm-results.html) @@ -21 +17 @@ You must choose a supported evaluator model to compute the metrics for your eval -You must provide a prompt dataset you want to use for the evaluation. For more information, see [Use a prompt dataset for a knowledge base evaluation in Amazon Bedrock](./knowledge-base-evaluation-prompt.html) +You must choose a supported evaluator model to compute the metrics for your evaluation. If you want to evaluate retrieval with response generation for an Amazon Bedrock knowledge base, then you must also choose a supported model for response generation. For more information, see [Prerequisites for creating RAG evaluations in Amazon Bedrock](./knowledge-base-evaluation-prereq.html) @@ -25 +21 @@ The following example shows you how to create a knowledge base evaluation job us -## Knowledge base evaluation jobs that use LLMs +## RAG evaluation jobs that use LLMs @@ -27 +23 @@ The following example shows you how to create a knowledge base evaluation job us -The following example shows you how to create a knowledge base evaluation job that uses Large Language Models (LLMs) for the evaluation. +The following example shows you how to create a RAG evaluation job that uses Large Language Models (LLMs) for the evaluation. @@ -36 +32 @@ _AWS Command Line Interface_ - --evaluation-context "RAG" \ + --application-type "RagEvaluation" \ @@ -92 +88 @@ _AWS Command Line Interface_ -_SDK for Python boto3_ +_AWS SDK for Python (Boto3)_ @@ -98 +94 @@ During preview, your AWS account management will provide you with a parameters f -The following python example demonstrates how to make a _Retrieve only_ boto3 API request. +The following python example demonstrates how to make a retrieve-only Boto3 API request. To learn more about creating an evaluation job using Boto3, refer to [`create_evaluation_job`](https://boto3.amazonaws.com/v1/documentation/api/1.35.6/reference/services/bedrock/client/create_evaluation_job.html) in the Boto3 documentation. @@ -104,2 +100,2 @@ The following python example demonstrates how to make a _Retrieve only_ boto3 AP - job_request = client.create_evaluation_job( - jobName="fkki-boto3-test1", + job_response = client.create_evaluation_job( + jobName="my_evaluation_job", @@ -108 +104 @@ The following python example demonstrates how to make a _Retrieve only_ boto3 AP - evaluationContext="RAG", + applicationType="RagEvaluation", @@ -163,2 +159,2 @@ The following python example demonstrates how to make a _Retrieve and generate_ - job_request = client.create_evaluation_job( - jobName="api-auto-job-titan", + job_response = client.create_evaluation_job( + jobName="my_evaluation_job", @@ -166,0 +163 @@ The following python example demonstrates how to make a _Retrieve and generate_ + applicationType="RagEvaluation", @@ -223,0 +221,120 @@ The following python example demonstrates how to make a _Retrieve and generate_ +## RAG evaluation jobs using your own inference response data + +To evaluate the performance of your own Retrieval Augmented Generation (RAG) sources, you can create a RAG evaluation job using your own inference response data. To learn how to create a prompt dataset with your own inference response data, see [Create a prompt dataset for a RAG evaluation in Amazon Bedrock](./knowledge-base-evaluation-prompt.html). The following sections show example AWS CLI commands and JSON files used to create retrieve-only and retrieve-and-generate evaluation jobs. + +### Create a retrieve-only evaluation job with your own inference response data + +Run the following AWS CLI command using the example JSON file. + + + aws bedrock create-evaluation-job --cli-input-json file://my_eval_job.json + + + { + "jobName": "my_rag_eval", + "roleArn": "arn:aws:iam::111122223333:role/service-role/Amazon-Bedrock-IAM-Role-Amazon-Bedrock-IAM-Role-20250218T063974", + "applicationType": "RagEvaluation", + "evaluationConfig": { + "automated": { + "datasetMetricConfigs": [ + { + "taskType": "General", + "dataset": { + "name": "text_dataset", + "datasetLocation": { + "s3Uri": "s3://amzn-s3-demo-bucket/input/retrieval-only/retrieve-eval-byoir.jsonl" + } + }, + "metricNames": [ + "Builtin.ContextCoverage", + "Builtin.ContextRelevance" + ] + } + ], + "evaluatorModelConfig": { + "bedrockEvaluatorModels": [ + { + "modelIdentifier": "us.meta.llama3-1-70b-instruct-v1:0" + } + ] + } + } + }, + "inferenceConfig": { + "ragConfigs": [ + { + "precomputedRagSourceConfig": { + "retrieveSourceConfig": { + "ragSourceIdentifier": "my_rag_source" + } + } + } + ] + }, + "outputDataConfig": { + "s3Uri": "s3://amzn-s3-demo-bucket/output/" + } + } + +### Create a retrieve-and-generate evaluation job with your own inference response data + +Run the following AWS CLI command using the example JSON file. + + + aws bedrock create-evaluation-job --cli-input-json file://my_eval_job.json + + + { + "jobName": "my_rag_eval", + "roleArn": "arn:aws:iam::111122223333:role/service-role/Amazon-Bedrock-IAM-Role-Amazon-Bedrock-IAM-Role-20250218T063974", + "applicationType": "RagEvaluation", + "evaluationConfig": { + "automated": { + "datasetMetricConfigs": [ + { + "taskType": "General", + "dataset": { + "name": "text_dataset", + "datasetLocation": { + "s3Uri": "s3://amzn-s3-demo-bucket/input/retrieve-and-generate/rng-eval-byoir.jsonl" + } + }, + "metricNames": [ + "Builtin.Correctness", + "Builtin.Completeness", + "Builtin.Helpfulness", + "Builtin.LogicalCoherence", + "Builtin.Faithfulness", + "Builtin.Harmfulness", + "Builtin.Stereotyping", + "Builtin.Refusal", + "Builtin.CitationCoverage", + "Builtin.CitationPrecision" + ] + } + ], + "evaluatorModelConfig": { + "bedrockEvaluatorModels": [ + { + "modelIdentifier": "us.meta.llama3-1-70b-instruct-v1:0" + } + ] + } + } + }, + "inferenceConfig": { + "ragConfigs": [ + { + "precomputedRagSourceConfig": { + "retrieveAndGenerateSourceConfig": { + "ragSourceIdentifier": "my_rag_source" + } + } + } + ] + }, + "outputDataConfig": { + "s3Uri": "s3://amzn-s3-demo-bucket/output/" + } + } +