AWS prescriptive-guidance: Remove natural language to query DSL pattern page
Summary
The entire prescriptive guidance pattern page 'Translate natural language into query DSL for OpenSearch and Elasticsearch queries' was deleted, including its summary, architecture, epics, IAM policy example, and prompt-engineering guidance. This appears to be a documentation removal/restructure rather than a content edit.
Security assessment
The diff only removes the whole page (all lines are deletions, no additions). While the removed content included an IAM policy with broad permissions (e.g. iam:*, aoss:*) and a 'Security enhancements' section, the change itself is a wholesale page deletion with no indication it fixes a specific vulnerability, CVE, or incident, and it removes rather than adds security documentation.
Evidence
-# Translate natural language into query DSL for OpenSearch and Elasticsearch queries
Diff
diff --git a/prescriptive-guidance/latest/patterns/translate-natural-language-query-dsl-opensearch-elasticsearch.md b/prescriptive-guidance/latest/patterns/translate-natural-language-query-dsl-opensearch-elasticsearch.md index f1179ec..8b13789 100644 --- a//prescriptive-guidance/latest/patterns/translate-natural-language-query-dsl-opensearch-elasticsearch.md +++ b//prescriptive-guidance/latest/patterns/translate-natural-language-query-dsl-opensearch-elasticsearch.md @@ -1 +0,0 @@ -[View a markdown version of this page](translate-natural-language-query-dsl-opensearch-elasticsearch.md) @@ -3,550 +1,0 @@ -[](/pdfs/prescriptive-guidance/latest/patterns/prescriptive-guidance.pdf#translate-natural-language-query-dsl-opensearch-elasticsearch "Open PDF") - -[Documentation](/index.html)[AWS Prescriptive Guidance](https://aws.amazon.com/prescriptive-guidance/)[Patterns](welcome.html) - -SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resourcesAdditional information - -# Translate natural language into query DSL for OpenSearch and Elasticsearch queries - - _Tabby Ward, Nicholas Switzer, and Breanne Warner, Amazon Web Services_ - -## Summary - -This pattern demonstrates how to use large language models (LLMs) to convert natural language queries into query domain-specific language (query DSL), which makes it easier for users to interact with search services such as OpenSearch and Elasticsearch without extensive knowledge of the query language. This resource is particularly valuable for developers and data scientists who want to enhance search-based applications with natural language querying capabilities, ultimately improving user experience and search functionality. - -The pattern illustrates techniques for prompt engineering, iterative refinement, and incorporation of specialized knowledge, all of which are crucial in synthetic data generation. Although this approach focuses primarily on query conversion, it implicitly demonstrates the potential for data augmentation and scalable synthetic data production. This foundation could be extended to more comprehensive synthetic data generation tasks, to highlight the power of LLMs in bridging unstructured natural language inputs with structured, application-specific outputs. - -This solution doesn't involve migration or deployment tools in the traditional sense. Instead, it focuses on demonstrating a proof of concept (PoC) for converting natural language queries to query DSL by using LLMs. - - * The pattern uses a Jupyter notebook as a step-by-step guide for setting up the environment and implementing the text-to-query conversion. - - * It uses Amazon Bedrock to access LLMs, which are crucial for interpreting natural language and generating appropriate queries. - - * The solution is designed to work with Amazon OpenSearch Service. You can follow a similar process for Elasticsearch, and the generated queries could potentially be adapted for similar search engines. - - - - -[Query DSL](https://opensearch.org/docs/latest/query-dsl/) is a flexible, JSON-based search language that's used to construct complex queries in both Elasticsearch and OpenSearch. It enables you to specify queries in the query parameter of search operations, and supports various query types. A DSL query includes leaf queries and compound queries. Leaf queries search for specific values in certain fields and encompass full-text, term-level, geographic, joining, span, and specialized queries. Compound queries act as wrappers for multiple leaf or compound clauses, and combine their results or modify their behavior. Query DSL supports the creation of sophisticated searches, ranging from simple, match-all queries to complex, multi-clause queries that produce highly specific results. Query DSL is particularly valuable for projects that require advanced search capabilities, flexible query construction, and JSON-based query structures. - -This pattern uses techniques such as few-shot prompting, system prompts, structured output, prompt chaining, context provision, and task-specific prompts for text-to-query DSL conversion. For definitions and examples of these techniques, see the Additional information section. - -## Prerequisites and limitations - -**Prerequisites** - -To effectively use the Jupyter notebook for converting natural language queries into query DSL queries, you need: - - * **Familiarity with Jupyter notebooks**. Basic understanding of how to navigate and run code in a Jupyter notebook environment. - - * **Python environment**. A working Python environment, preferably Python 3.x, with the necessary libraries installed. - - * **Elasticsearch or OpenSearch knowledge**. Basic knowledge of Elasticsearch or OpenSearch, including its architecture and how to perform queries. - - * **AWS account**. An active AWS account to access Amazon Bedrock and other related services. - - * **Libraries and dependencies**. Installation of specific libraries mentioned in the notebook, such as `boto3` for AWS interaction, and any other dependencies required for LLM integration. - - * **Model access within Amazon Bedrock**. This pattern uses three Claude LLMs from Anthropic. Open the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/) and choose **Model access**. On the next screen, choose **Enable specific models** and select these three models: - - * Claude 3 Sonnet - - * Claude 3.5 Sonnet - - * Claude 3 Haiku - - * **Proper IAM policies and IAM role**. To run the notebook in an AWS account, your AWS Identity and Access Management (IAM) role requires the `SagemakerFullAccess` policy as well as the policy that’s provided in the Additional information section, which you can name `APGtext2querydslpolicy`. This policy includes subscribing to the three Claude models listed. - - - - -Having these prerequisites in place ensures a smooth experience when you work with the notebook and implement the text-to-query functionality. - -**Limitations** - - * **Proof of concept status**. This project is primarily intended as a proof of concept (PoC). It demonstrates the potential of using LLMs to convert natural language queries into query DSL, but it might not be fully optimized or production-ready. - - * **Model limitations** : - -**Context window constraints** _._ When using the LLMs that are available on Amazon Bedrock, be aware of the context window limitations: - -Claude models (as of September 2024): - - * Claude 3 Opus: 200,000 tokens - - * Claude 3 Sonnet: 200,000 tokens - - * Claude 3 Haiku: 200,000 tokens - -Other models on Amazon Bedrock might have different context window sizes. Always check the latest documentation for the latest information. - -**Model availability** _._ The availability of specific models on Amazon Bedrock can vary. Make sure that you have access to the required models before you implement this solution. - - * **Additional limitations** - - * **Query complexity**. The effectiveness of the natural language to query DSL conversion might vary depending on the complexity of the input query. - - * **Version compatibility**. The generated query DSL might need adjustments based on the specific version of Elasticsearch or OpenSearch that you use. - - * **Performance**. This pattern provides a PoC implementation, so query generation speed and accuracy might not be optimal for large-scale production use. - - * **Cost**. Using LLMs in Amazon Bedrock incurs costs. Be aware of the pricing structure for your chosen model. For more information, see [Amazon Bedrock pricing](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-pricing.html). - - * **Maintenance**. Regular updates to the prompts and model selection might be necessary to keep up with advancements in LLM technology and changes in query DSL syntax. - - - - -**Product versions** - -This solution was tested in Amazon OpenSearch Service. If you want to use Elasticsearch, you might have to make some changes to replicate the exact functionality of this pattern. - - * **OpenSearch version compatibility**.**** OpenSearch maintains backward compatibility within major versions. For example: - - * OpenSearch 1.x clients are generally compatible with OpenSearch 1.x clusters. - - * OpenSearch 2.x clients are generally compatible with OpenSearch 2.x clusters. - -However, it's always best to use the same minor version for both client and cluster when possible. - - * **OpenSearch API compatibility**.**** OpenSearch maintains API compatibility with Elasticsearch OSS 7.10.2 for most operations. However, some differences exist, especially in newer versions. - - * **OpenSearch upgrade considerations** : - - * Direct downgrades are not supported. Use snapshots for rollback if needed. - - * When you upgrade, check the [compatibility matrix and release notes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html) for any breaking changes. - - - - -**Elasticsearch considerations** - - * **Elasticsearch version**. The major version of Elasticsearch you're using is crucial, because query syntax and features can change between major versions. Currently, the latest stable version is Elasticsearch 8.x. Make sure that your queries are compatible with your specific Elasticsearch version. - - * **Elasticsearch query DSL library version**. If you're using the Elasticsearch query DSL Python library, make sure that its version matches your Elasticsearch version. For example: - - * For Elasticsearch 8.x, use an `elasticsearch-dsl` version that's greater or equal to 8.0.0 but smaller than 9.0.0. - - * For Elasticsearch 7.x, use an `elasticsearch-dsl` version that's greater or equal to 7.0.0 but smaller than 8.0.0. - - * **Client library version**. Whether you're using the official Elasticsearch client or a language-specific client, make sure that it's compatible with your Elasticsearch version. - - * **Query DSL version**. Query DSL evolves with Elasticsearch versions. Some query types or parameters might be deprecated or introduced in different versions. - - * **Mapping version**. The way you define mappings for your indexes and change between versions. Always check the mapping documentation for your specific Elasticsearch version. - - * **Analysis tools versions**. If you're using analyzers, tokenizers, or other text analysis tools, their behavior or availability might change between versions. - - - - -## Architecture - -**Target architecture** - -The following diagram illustrates the architecture for this pattern. - - - -where: - - 1. User input and system prompt with few-shot prompting examples. The process begins with a user who provides a natural language query or a request for schema generation. - - 2. Amazon Bedrock. The input is sent to Amazon Bedrock, which serves as the interface to access the Claude LLM. - - 3. Claude 3 Sonnet LLM. Amazon Bedrock uses Claude 3 Sonnet from the Claude 3 family of LLMs to process the input. It interprets and generates the appropriate Elasticsearch or OpenSearch query DSL. For schema requests, it generates synthetic Elasticsearch or OpenSearch mappings. - - 4. Query DSL generation. For natural language queries, the application takes the LLM's output and formats it into a valid Elasticsearch or OpenSearch Service query DSL. - - 5. Synthetic data generation. The application also takes schemas to create synthetic Elasticsearch or OpenSearch data to be loaded into an OpenSearch Serverless collection for testing. - - 6. OpenSearch or Elasticsearch. The generated Query DSL is queried against an OpenSearch Serverless collection on all indexes. JSON output contains the relevant data and number of _hits_ from the data that resides in the OpenSearch Serverless collection. - - - - -**Automation and scale** - -The code that's provided with this pattern is built strictly for PoC purposes. The following list provides a few suggestions for automating and scaling the solution further and moving the code to production. These enhancements are outside the scope of this pattern. - - * Containerization: - - * Dockerize the application to ensure consistency across different environments. - - * Use container orchestration platforms such as Amazon Elastic Container Service (Amazon ECS) or Kubernetes for scalable deployments. - - * Serverless architecture: - - * Convert the core functionality into AWS Lambda functions. - - * Use Amazon API Gateway to create RESTful endpoints for the natural language query input. - - * Asynchronous processing: - - * Implement Amazon Simple Queue Service (Amazon SQS) to queue incoming queries. - - * Use AWS Step Functions to orchestrate the workflow of processing queries and generating query DSL. - - * Caching: - - * Implement a mechanism to cache the prompts. - - * Monitoring and logging: