AWS opensearch-service documentation change
Summary
Restructured documentation to introduce NextGen vector search collections, updated tutorials for NextGen workflows, added Classic vector collection section, revised limitations, and removed billion-scale workloads section.
Security assessment
Changes focus on feature enhancements (NextGen optimizations, compression levels, GPU acceleration) and documentation reorganization. No security vulnerabilities, patches, or incident responses are mentioned. Security references (encryption, IAM) are existing content.
Diff
diff --git a/opensearch-service/latest/developerguide/serverless-vector-search.md b/opensearch-service/latest/developerguide/serverless-vector-search.md index d5d6856c1..97d2e5d03 100644 --- a//opensearch-service/latest/developerguide/serverless-vector-search.md +++ b//opensearch-service/latest/developerguide/serverless-vector-search.md @@ -7 +7 @@ -Getting startedFiltered searchBillion scale workloadsLimitationsNext steps +NextGen vector search collectionsGetting startedFiltered searchLimitationsWorking with Classic vector collectionsNext steps @@ -11 +11 @@ Getting startedFiltered searchBillion scale workloadsLimitationsNext steps -The _vector search_ collection type in OpenSearch Serverless provides a similarity search capability that is scalable and high performing. It makes it easy for you to build modern machine learning (ML) augmented search experiences and generative artificial intelligence (AI) applications without having to manage the underlying vector database infrastructure. +With the _vector search_ collection type in OpenSearch Serverless, you can perform scalable, high-performing similarity searches. You can build modern machine learning (ML) augmented search experiences and generative artificial intelligence (AI) applications without managing the underlying vector database infrastructure. @@ -15 +15 @@ Use cases for vector search collections include image searches, document searche -Because the vector engine for OpenSearch Serverless is powered by the [k-nearest neighbor (k-NN) search feature](https://opensearch.org/docs/latest/search-plugins/knn/index/) in OpenSearch, you get the same functionality with the simplicity of a serverless environment. The engine supports [k-NN plugin API](https://opensearch.org/docs/latest/search-plugins/knn/api/). With these operations, you can take advantage of full-text search, advanced filtering, aggregations, geospatial queries, nested queries for faster retrieval of data, and enhanced search results. +The vector engine for OpenSearch Serverless uses the [k-nearest neighbor (k-NN) search feature](https://opensearch.org/docs/latest/search-plugins/knn/index/) in OpenSearch. You get the same functionality with the simplicity of a serverless environment. The engine supports the [k-NN plugin API](https://opensearch.org/docs/latest/search-plugins/knn/api/). With these operations, you can use full-text search, advanced filtering, aggregations, geospatial queries, and nested queries for faster data retrieval and enhanced search results. @@ -19 +19,9 @@ The vector engine provides distance metrics such as Euclidean distance, cosine s -###### Note +## NextGen vector search collections + +NextGen vector search scales on demand based on workload to optimize the balance between cost and performance. Only the data blocks required to serve active search requests are loaded into memory, and workers scale dynamically based on required memory and CPU resources. When the collection is idle with no ongoing requests, both indexing and search scale to zero, providing additional cost savings. By default, NextGen includes built-in optimizations that improve recall while reducing cost and latency. + + * **Custom doc ID** – Custom document IDs are supported in NextGen collections, making it easier for customers to perform updates or index documents with user-provided IDs. + + * **32x compression indices** – All indexes are created with advanced 32x compression technique by default. You can override the default compression level and select any supported compression level: 1x, 2x, 8x, 16x, or 32x (default). + + * **Index build acceleration** – GPU acceleration is enabled by default to help build large scale vector indexes faster and more efficiently. It reduces the time needed to index data into vector indexes, providing a high throughput indexing experience and cost savings. GPU resources are provisioned only when needed during index build operations. You can control GPU usage on a per index basis using the setting `index.knn.remote_index_build.enabled`. For more information, see [GPU-acceleration for vector indexing](./gpu-acceleration-vector-index.html). @@ -21 +29 @@ The vector engine provides distance metrics such as Euclidean distance, cosine s -Note the following information: + * **Simplified API** – NextGen vector search collections do not require the `engine` and `mode` parameters in index mappings. The system automatically determines the optimal configuration internally, reducing the complexity of index creation. @@ -23 +31 @@ Note the following information: - * Amazon OpenSearch Serverless supports Faiss 16-bit scalar quantization which can be used to perform conversions between 32-bit floating and 16-bit vectors. To learn more, see [ Faiss 16-bit scalar quantization](https://opensearch.org/docs/latest/search-plugins/knn/knn-vector-quantization/#faiss-16-bit-scalar-quantization). You can also use binary vectors to reduce memory costs. For more information, see [Binary vectors](https://opensearch.org/docs/latest/field-types/supported-field-types/knn-vector#binary-vectors). + * **Optimized search response** – By default, search responses in NextGen vector collections exclude the original vector from the results. This reduces end to end search latency and response payload size. To include vectors in the search response, see Retrieve full document with vectors. @@ -25 +33 @@ Note the following information: - * Amazon OpenSearch Serverless supports disk-based vector search. Disk-based vector search significantly reduces the operational costs for vector workloads in low-memory environments. For more information, see [Disk-based vector search](https://docs.opensearch.org/2.19/vector-search/optimizing-storage/disk-based-vector-search/). + * NextGen vector collections have a read-after-write latency (`refresh_interval`) of 10 seconds. @@ -85 +93 @@ For more information about OpenSearch Serverless IAM permissions, see [Identity -A _collection_ is a group of OpenSearch indexes that work together to support a specific workload or use case. +To create a new collection, follow the unified collection creation flow (NextGen Express Create), which automatically configures encryption, network, and data access policies. For instructions, see [Create a NextGen collection (Express Create)](./serverless-create.html#serverless-create-nextgen-easy). @@ -87 +95 @@ A _collection_ is a group of OpenSearch indexes that work together to support a -###### To create an OpenSearch Serverless collection +For the rest of this tutorial, the example collection is named `housing` and is a NextGen vector search collection. @@ -89 +97 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). +###### Note @@ -91 +99 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 2. Choose **Collections** in the left navigation pane and choose **Create collection**. +If you choose to create a Classic vector collection instead, see Working with Classic vector collections for procedures specific to Classic collections. @@ -93 +101 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 3. Name the collection **housing**. +### Step 3: Upload and search data @@ -95 +103 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 4. For collection type, choose **Vector search**. For more information, see [Choosing a collection type](./serverless-overview.html#serverless-usecase). +An _index_ is a collection of documents with a common data schema that provides a way for you to store, search, and retrieve your vector embeddings and other fields. You can create and upload data to indexes in an OpenSearch Serverless collection by using the [Dev Tools](https://opensearch.org/docs/latest/dashboards/dev-tools/index-dev/) console in OpenSearch Dashboards, or an HTTP tool such as [Postman](https://www.postman.com/downloads/) or [awscurl](https://github.com/okigan/awscurl). This tutorial uses Dev Tools. For programmatic access using the Python SDK, see [Ingesting data into Amazon OpenSearch Serverless collections](./serverless-clients.html). @@ -97 +105 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 5. Under **Deployment type** , clear **Enable redundancy (active replicas)**. This creates a collection in development or testing mode, and reduces the number of OpenSearch Compute Units (OCUs) in your collection to two. If you want to create a production environment in this tutorial, leave the check box selected. +###### To index and search data in the housing collection @@ -99 +107,26 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 6. Under **Security** , select **Easy create** to streamline your security configuration. All the data in the vector engine is encrypted in transit and at rest by default. The vector engine supports fine-grained IAM permissions so that you can define who can create, update, and delete encryptions, networks, collections, and indexes. + 1. To create an index for your new collection, send the following request in the [Dev Tools](https://opensearch.org/docs/latest/dashboards/dev-tools/index-dev/) console. By default, this creates an index with Euclidean distance and 32x compression. + + PUT housing-index + { + "settings": { + "index.knn": true + }, + "mappings": { + "properties": { + "housing-vector": { + "type": "knn_vector", + "dimension": 3, + "space_type": "l2" + }, + "title": { + "type": "text" + }, + "price": { + "type": "long" + }, + "location": { + "type": "geo_point" + } + } + } + } @@ -101 +134 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 7. Choose **Next**. + 2. To use a different compression level, set `compression_level` in the field mapping. The following example creates an index with `compression_level` set to 1x. @@ -103 +136,25 @@ A _collection_ is a group of OpenSearch indexes that work together to support a - 8. Review your collection settings and choose **Submit**. Wait several minutes for the collection status to become `Active`. + PUT housing-index-1x + { + "settings": { + "index.knn": true + }, + "mappings": { + "properties": { + "housing-vector": { + "type": "knn_vector", + "dimension": 3, + "compression_level": "1x", + "space_type": "l2" + }, + "title": { + "type": "text" + }, + "price": { + "type": "long" + }, + "location": { + "type": "geo_point" + } + } + } + } @@ -104,0 +162 @@ A _collection_ is a group of OpenSearch indexes that work together to support a +Supported compression levels are 1x, 2x, 8x, 16x, and 32x. @@ -105,0 +164 @@ A _collection_ is a group of OpenSearch indexes that work together to support a + 3. To index documents into `housing-index`, you can use a system-generated ID (POST) or a user-provided ID (PUT). @@ -106,0 +166,8 @@ A _collection_ is a group of OpenSearch indexes that work together to support a + # System-generated document ID + POST housing-index/_doc + { + "housing-vector": [10, 20, 30], + "title": "2 bedroom in downtown Seattle", + "price": "2800", + "location": "47.71, 122.00" + } @@ -108 +175,8 @@ A _collection_ is a group of OpenSearch indexes that work together to support a -### Step 3: Upload and search data + # User-provided document ID + PUT housing-index/_doc/100 + { + "housing-vector": [10, 20, 30], + "title": "2 bedroom in downtown Seattle", + "price": "2800", + "location": "47.71, 122.00" + } @@ -110 +184,121 @@ A _collection_ is a group of OpenSearch indexes that work together to support a -An _index_ is a collection of documents with a common data schema that provides a way for you to store, search, and retrieve your vector embeddings and other fields. You can create and upload data to indexes in an OpenSearch Serverless collection by using the [Dev Tools](https://opensearch.org/docs/latest/dashboards/dev-tools/index-dev/) console in OpenSearch Dashboards, or an HTTP tool such as [Postman](https://www.postman.com/downloads/) or [awscurl](https://github.com/okigan/awscurl). This tutorial uses Dev Tools. For programmatic access using the Python SDK, see [Ingesting data into Amazon OpenSearch Serverless collections](./serverless-clients.html). + 4. To search for properties similar to the ones in your index, send the following query. By default, the search response excludes the original vector from `_source` to reduce latency and payload size. + + GET housing-index/_search + { + "size": 5, + "query": { + "knn": { + "housing-vector": { + "vector": [10, 20, 30], + "k": 2 + } + } + } + } + +The response excludes the `housing-vector` field from `_source`: + + { + "took": 10, + "timed_out": false, + "_shards": { + "total": 0, + "successful": 0, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "housing-index", + "_id": "100", + "_score": 1, + "_source": { + "price": "2800", + "location": "47.71, 122.00", + "title": "2 bedroom in downtown Seattle" + } + } + ] + } + } + + + + +#### Retrieve full document with vectors + +To override the default behavior, set `_source` to `true` in the search request. You can also use the `includes`/`excludes` options of `_source` to retrieve specific fields. + + + GET housing-index/_search + { + "size": 5, + "_source": true, + "query": { + "knn": { + "housing-vector": { + "vector": [10, 20, 30],