AWS bedrock documentation change
Summary
Updated OpenSearch requirements, added managed cluster configuration details, expanded Titan V2 dimensions, restructured Pinecone setup instructions, and added security-related IAM/KMS documentation
Security assessment
Added documentation about IAM permissions requirements, encryption (KMS for secrets), and recommendations for fine-grained access control. These are security best practices but don't address a specific disclosed vulnerability.
Diff
diff --git a/bedrock/latest/userguide/knowledge-base-setup.md b/bedrock/latest/userguide/knowledge-base-setup.md index 2e86144d9..d0cb572ab 100644 --- a/bedrock/latest/userguide/knowledge-base-setup.md +++ b/bedrock/latest/userguide/knowledge-base-setup.md @@ -5 +5 @@ -# Prerequisites for your own vector store for a knowledge base +# Prerequisites for using a vector store you created for a knowledge base @@ -9 +9,5 @@ To store the vector embeddings that your documents are converted to, you use a v -If you want to store binary vector embeddings instead of the standard floating-point (float32) vector embeddings, then you must use a vector store that supports binary vectors. Amazon OpenSearch Serverless is currently the only vector store that supports storing binary vectors. +If you want to store binary vector embeddings instead of the standard floating-point (float32) vector embeddings, then you must use a vector store that supports binary vectors. + +###### Note + +Amazon OpenSearch Serverless and Amazon OpenSearch Managed clusters are the only vector stores that support storing binary vectors. @@ -56 +60 @@ Titan G1 Embeddings - Text | 1,536 -Titan V2 Embeddings - Text | 1,024 +Titan V2 Embeddings - Text | 1,024, 512, and 256 @@ -85,0 +90,116 @@ For more detailed documentation on setting up a vector store in Amazon OpenSearc +Amazon OpenSearch Service Managed Clusters + + +###### Important + + * Before using any domain resources in OpenSearch Managed clusters, you need to configure certain IAM access permissions and policies. For more information, see [Prerequisites and permissions required for using OpenSearch Managed Clusters with Amazon Bedrock Knowledge Bases](./kb-osm-permissions-prereq.html). + + * If you encounter data ingestion failures, it might indicate insufficient OpenSearch domain capacity. To resolve this issue, increase your domain's capacity by provisioning higher IOPS and by increasing the throughput settings. For more information, see [Operational best practices for Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html). + + + + + 1. To create a domain and vector index in OpenSearch Cluster in the AWS Management Console, follow the steps described in [Creating and managing OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) in the _Amazon OpenSearch Service Developer Guide_. + +Note the following considerations while setting up your domain: + + 1. Give the domain a name of your choice. + + 2. We recommend that you use the **Easy create** option to get started quickly with creating your domain. + +###### Note + +This option gives you a domain with a low throughput. If you have larger workloads that require a higher throughput, choose the **Standard Create** option. You can adjust the capacity later as required. With this option, you can start with the lowest capacity, which can then be modified later as needed. + + 3. For Network, you must choose **Public access**. OpenSearch domains that are behind a VPC are not supported for your Knowledge Base. + + 4. For **Version** , if you're using binary vector embeddings, Amazon Bedrock Knowledge Bases requires an Engine version of 2.16 or later. In addition, a version of 2.13 or higher is required to create a k-nn index. For more information, see [K-NN Search](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/knn.html) in the _Amazon OpenSearch Service developer guide_. + + 5. We recommend that you use the **Dual-stack mode**. + + 6. We recommend that you enable **Fine-grained access control** to protect the data in your domain, and further control the permissions that grants your Knowledge base service role access to the OpenSearch domain and make requests. + + 7. Leave all other settings to their default values and choose **Create** to create your domain. + + 2. Once the domain is created, click it to take note of the **Domain ARN** and **Domain endpoint** for when you create the knowledge base. + + 3. After you've created the domain, you can create a vector index by running the following commands on an OpenSearch dashboard or using curl commands. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/latest/search-plugins/knn/knn-index/). + +When running the command: + + * Provide a name for the vector field (for example, `embeddings`). + + * Make sure that the vector used for search is **faiss**. **nmslib** is not supported. + + * For the number of dimensions in the vector, refer to the following table to determine how many dimensions the vector should contain: + +###### Note + +The Titan V2 Embeddings - Text model supports multiple dimensions. It can also be 256 or 512. + +Model | Dimensions +---|--- +Titan G1 Embeddings - Text | 1,536 +Titan V2 Embeddings - Text | 1,024, 512, and 256 +Cohere Embed English | 1,024 +Cohere Embed Multilingual | 1,024 + + * You can add two fields to configure the vector index to store additional metadata that a knowledge base can retrieve with vectors. The following table describes the fields and the values to specify for each of them. + +Field description | Mapping field +---|--- +Amazon Bedrock chunks the raw text from your data and stores the chunks in this field. | Specified as an object, for example, `AMAZON_BEDROCK_TEXT_CHUNK`. +Amazon Bedrock stores metadata related to your knowledge base in this field. | Specified as an object, for example, `AMAZON_BEDROCK_METADATA`. + + PUT /<index-name> + { + "settings": { + "index": { + "knn": true + } + }, + "mappings": { + "properties": { + "<vector-name>": { + "type": "knn_vector", + "dimension": <embedding-dimension>, + "data_type": "binary", # Only needed for binary embeddings + "space_type": "l2" | "hamming", # Use l2 for float embeddings and hamming for binary embeddings + "method": { + "name": "hnsw", + "engine": "faiss", + "parameters": { + "ef_construction": 128, + "m": 24 + } + } + }, + + "AMAZON_BEDROCK_METADATA": { + "type": "text", + "index": "false" + }, + "AMAZON_BEDROCK_TEXT_CHUNK": { + "type": "text", + "index": "true" + } + } + } + } + + 4. Take note of the domain ARN and endpoint, and the names you choose for the vector index name, vector field name, and metadata management mapping field names for when you create your knowledge base. + + + + +After the vector index is created, you can proceed to [create your knowledge base](./knowledge-base-create.html). The following table summarizes where you will enter each piece of information that you took note of. + +Field | Corresponding field in knowledge base setup (Console) | Corresponding field in knowledge base setup (API) | Description +---|---|---|--- +Domain ARN | Domain ARN | domainARN | The Amazon Resource Name (ARN) of the OpenSearch domain. +Domain endpoint | Domain endpoint | domainEndpoint | The endpoint to connect to the OpenSearch domain. +Vector index name | Vector index name | vectorIndexName | The name of the vector index. +Vector field name | Vector field | vectorField | The name of the field in which to store vector embeddings for your data sources. +Metadata management (first mapping field) | Text field | textField | The name of the field in which to store the raw text from your data sources. +Metadata management (second mapping field) | Bedrock-managed metadata field | metadataField | The name of the field in which to store metadata that Amazon Bedrock manages. + @@ -144 +264 @@ Titan G1 Embeddings - Text | 1,536 -Titan V2 Embeddings - Text | 1,024 +Titan V2 Embeddings - Text | 1,024, 512, and 256 @@ -185 +305 @@ While you set up the vector store, take note of the following information, which - * **Name Space** – (Optional) The namespace to be used to write new data to your database. For more information, see [Using namespaces](https://docs.pinecone.io/docs/namespaces). + * **Credentials secret ARN** – The Amazon Resource Name (ARN) of the secret that you created in AWS Secrets Manager that contains the username and password for a database user. @@ -186,0 +307 @@ While you set up the vector store, take note of the following information, which + * **(Optional) Customer-managed KMS key for your Credentials secret ARN** – if you encrypted your credentials secret ARN, provide the KMS key so that Amazon Bedrock can decrypt it. @@ -187,0 +309 @@ While you set up the vector store, take note of the following information, which + * **Name Space** – (Optional) The namespace to be used to write new data to your database. For more information, see [Using namespaces](https://docs.pinecone.io/docs/namespaces). @@ -190 +311,0 @@ While you set up the vector store, take note of the following information, which -There are additional configurations that you must provide when creating a Pinecone index: @@ -192 +312,0 @@ There are additional configurations that you must provide when creating a Pineco - * **Name** – The name of the vector index. Choose any valid name of your choice. Later, when you create your knowledge base, enter the name you choose in the **Vector index name** field. @@ -194 +314 @@ There are additional configurations that you must provide when creating a Pineco - * **Dimensions** – The number of dimensions in the vector. Refer to the following table to determine how many dimensions the vector should contain. +There are additional configurations that you must provide when creating a Pinecone index: @@ -196,6 +316 @@ There are additional configurations that you must provide when creating a Pineco -Model | Dimensions ----|--- -Titan G1 Embeddings - Text | 1,536 -Titan V2 Embeddings - Text | 1,024 -Cohere Embed English | 1,024 -Cohere Embed Multilingual | 1,024 + * **Text field name** – The name of the field which Amazon Bedrock should store the raw chunk text in. @@ -203 +318 @@ Cohere Embed Multilingual | 1,024 - * **Distance metric** – The metric used to measure the similarity between vectors. We recommend that you experiment with different metrics for your use-case. We recommend starting with **cosine similarity**. + * **Metadata field name** – The name of the field which Amazon Bedrock should store source attribution metadata in. @@ -245 +360 @@ Titan G1 Embeddings - Text | 1,536 -Titan V2 Embeddings - Text | 1,024 +Titan V2 Embeddings - Text | 1,024, 512, and 256 @@ -326 +441 @@ Prerequisites for your knowledge base data -Create a knowledge base +Prerequisites for OpenSearch Managed Clusters