AWS opensearch-service documentation change
Summary
Restructured observability documentation to focus on workflow stages (collection, ingestion, analytics), added OpenSearch UI vs Dashboards comparison, and removed specific PPL query examples in favor of architectural guidance
Security assessment
The changes focus on workflow explanation, tool comparisons, and architectural guidance without mentioning security controls, vulnerabilities, or security-specific features. While discussing data collection and pipelines, there's no reference to encryption, access control, or other security mechanisms.
Diff
diff --git a/opensearch-service/latest/developerguide/observability.md b/opensearch-service/latest/developerguide/observability.md index 78c4322ca..a7f791840 100644 --- a//opensearch-service/latest/developerguide/observability.md +++ b//opensearch-service/latest/developerguide/observability.md @@ -5 +5 @@ -Explore your data with event analyticsCreate visualizationsDive deeper with Trace Analytics +The observability workflow in OpenSearch ServiceOpenSearch UI and OpenSearch Dashboards @@ -9 +9 @@ Explore your data with event analyticsCreate visualizationsDive deeper with Trac -The default installation of OpenSearch Dashboards for Amazon OpenSearch Service includes the Observability plugin, which you can use to visualize data-driven events using Piped Processing Language (PPL) in order to explore, discover, and query data stored in OpenSearch. The plugin requires OpenSearch 1.2 or later. +**Observability** is the practice of gaining insight into the internal state and performance of your complex systems by examining their outputs. Traditional monitoring can tell you that your system is down, observability helps you understand why it's down by allowing you to ask new questions about your data. @@ -11 +11 @@ The default installation of OpenSearch Dashboards for Amazon OpenSearch Service -The Observability plugin provides a unified experience for collecting and monitoring metrics, logs, and traces from common data sources. Data collection and monitoring in one place enables full-stack, end-to-end observability of your entire infrastructure. +Amazon OpenSearch Service provides a unified solution by collecting and correlating key types of telemetry data. @@ -13 +13 @@ The Observability plugin provides a unified experience for collecting and monito -###### Note + * **Logs** provide timestamped records of events, such as application errors, user requests, or system status messages. @@ -15 +15 @@ The Observability plugin provides a unified experience for collecting and monito -This documentation provides a brief overview of Observability in OpenSearch Service. For comprehensive documentation of the Observability plugin, including permissions, see [Observability](https://opensearch.org/docs/latest/observing-your-data/). + * **Traces** represent the end-to-end journey of a request as it travels through the different services in a distributed system. @@ -17 +16,0 @@ This documentation provides a brief overview of Observability in OpenSearch Serv -Everyone's process for exploring data is different. If you’re new to exploring data and creating visualizations, we recommend trying a workflow like the following. @@ -19 +17,0 @@ Everyone's process for exploring data is different. If you’re new to exploring -## Explore your data with event analytics @@ -21 +18,0 @@ Everyone's process for exploring data is different. If you’re new to exploring -To start, let's say that you're collecting flight data in your OpenSearch Service domain and you want to find out which airline had the most flights arriving in Pittsburgh International Airport last month. You write the following PPL query: @@ -22,0 +20 @@ To start, let's say that you're collecting flight data in your OpenSearch Servic +By bringing these data types together, Amazon OpenSearch Service helps operations teams, site reliability engineers, and developers detect, diagnose, and resolve operational issues faster. @@ -24,3 +22 @@ To start, let's say that you're collecting flight data in your OpenSearch Servic - source=opensearch_dashboards_sample_data_flights | - stats count() by Dest, Carrier | - where Dest = "Pittsburgh International Airport" +## The observability workflow in OpenSearch Service @@ -28 +24 @@ To start, let's say that you're collecting flight data in your OpenSearch Servic -This query pulls data from the index named `opensearch_dashboards_sample_data_flights`. It then uses the `stats` command to get a total count of flights and groups it according to destination airport and carrier. Finally, it uses the `where` clause to filter the results to flights arriving in Pittsburgh International Airport. +Getting data from your applications into OpenSearch Service for analysis uses a multi-stage pipeline: @@ -30 +26 @@ This query pulls data from the index named `opensearch_dashboards_sample_data_fl -Here's what the data looks like when displayed over the last month: + @@ -32 +28 @@ Here's what the data looks like when displayed over the last month: - + 1. **Collection** @@ -34 +30 @@ Here's what the data looks like when displayed over the last month: -You can choose the **PPL** button in the query editor to get usage information and examples for each PPL command: +The process begins at the source with lightweight agents or collectors that gather telemetry data from your signal producers such as applications and infrastructure. Two common open-source agents are: @@ -36 +32 @@ You can choose the **PPL** button in the query editor to get usage information a - + * [OpenTelemetry](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-otel.html) (OTel) collector – The industry standard and preferred method for collecting logs and traces. @@ -38 +34 @@ You can choose the **PPL** button in the query editor to get usage information a -Let's look at a more complex example, which queries for information about flight delays: + * [Fluent Bit](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-fluentbit.html) – A lightweight, high-performance log processor and forwarder that supports the OTel schema for logs and traces. @@ -39,0 +36 @@ Let's look at a more complex example, which queries for information about flight + 2. **Ingestion (Amazon OpenSearch Ingestion)** @@ -41,5 +38 @@ Let's look at a more complex example, which queries for information about flight - source=opensearch_dashboards_sample_data_flights | - where FlightDelayMin > 0 | - stats sum(FlightDelayMin) as minimum_delay, count() as total_delayed by Carrier, Dest | - eval avg_delay=minimum_delay / total_delayed | - sort - avg_delay +After telemetry is collected, data is sent to OpenSearch Ingestion, a fully managed, serverless data pipeline. You can create custom pipelines to: @@ -47 +40 @@ Let's look at a more complex example, which queries for information about flight -Each command in the query impacts the final output: + * **Filter** – Remove unnecessary data to reduce storage costs. @@ -49 +42 @@ Each command in the query impacts the final output: - * `source=opensearch_dashboards_sample_data_flights` \- pulls data from the same index as the previous example + * **Enrich** – Add valuable metadata, such as geographic information from an IP address. @@ -51 +44 @@ Each command in the query impacts the final output: - * `where FlightDelayMin > 0` \- filters the data to flights that were delayed + * **Transform and normalize** – Structure unstructured logs into a consistent format. @@ -53 +46 @@ Each command in the query impacts the final output: - * `stats sum(FlightDelayMin) as minimum_delay, count() as total_delayed by Carrier` \- for each carrier, gets the total minimum delay time and the total count of delayed flights + * **Route** – Send different types of data to different OpenSearch Service indexes or Amazon S3. @@ -55 +48 @@ Each command in the query impacts the final output: - * `eval avg_delay=minimum_delay / total_delayed` \- calculates the average delay time for each carrier by dividing the minimum delay time by the total number of delayed flights + 3. **Analytics and visualization** @@ -57 +50 @@ Each command in the query impacts the final output: - * `sort - avg_delay` \- sorts the results by average delay in descending order +After processing, data is loaded into an OpenSearch Service domain or collection. You can store, index, and analyze vast amounts of data in near real time. You interact with this data through a visualization interface such as [OpenSearch UI's observability workspace](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/application.html) to run queries, build dashboards, and set up alerts. @@ -58,0 +52 @@ Each command in the query impacts the final output: + @@ -62 +55,0 @@ Each command in the query impacts the final output: -With this query, you can determine that OpenSearch Dashboards Airlines has, on average, fewer delays. @@ -64 +57 @@ With this query, you can determine that OpenSearch Dashboards Airlines has, on a - +## OpenSearch UI and OpenSearch Dashboards @@ -66 +59 @@ With this query, you can determine that OpenSearch Dashboards Airlines has, on a -You can find more sample PPL queries under **Queries and Visualizations** on the **Event analytics** page. +OpenSearch Service provides two distinct user interfaces for observability. We recommend that you use OpenSearch UI and set up an [observability workspace](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/application.html) for new installations and migrate from existing OpenSearch Dashboards. Below is a table outlining the benefits of OpenSearch UI v. traditional OpenSearch Dashboards. @@ -68 +61,6 @@ You can find more sample PPL queries under **Queries and Visualizations** on the -## Create visualizations +Feature | OpenSearch UI | OpenSearch Dashboards +---|---|--- +Data sources | **Multi-source** – can connect to multiple OpenSearch Service domains, OpenSearch Serverless collections, and other data sources. | **Single-source** – co-located with a single OpenSearch Service domain. +Updates | New features arrive here first because it is not tied to a specific OpenSearch version. | New features are tied to the OpenSearch version. Updates may be deprecated in the future. +Availability | Hosted in the AWS Cloud ensuring zero downtime during cluster upgrades. | Can become temporarily unavailable during domain maintenance and upgrades +Collaboration | Features workspaces for curated team collaboration on specific workflows. | Collaboration is based on sharing saved objects in a single domain. @@ -70 +68 @@ You can find more sample PPL queries under **Queries and Visualizations** on the -Once you correctly query the data that you're interested in, you can save those queries as visualizations: +**Note** – To make getting started easier, we’ve created a new [Get Started](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-otel.html) workflow for logs in the Amazon OpenSearch Service console which will set up a new OTel tailored ingestion pipeline, allow you to select an existing OpenSearch cluster, and create a new OpenSearch UI application with an observability workspace created. All you have to do is point your OTel agents to the new ingestion endpoint and you are ready to unlock insights on your OTel formatted data. @@ -72,9 +70 @@ Once you correctly query the data that you're interested in, you can save those - - -Then add those visualizations to [operational panels](https://opensearch.org/docs/latest/observability-plugin/operational-panels) to compare different pieces of data. Leverage [notebooks](https://opensearch.org/docs/latest/observability-plugin/notebooks) to combine different visualizations and code blocks that you can share with team members. - -## Dive deeper with Trace Analytics - -[Trace Analytics](./trace-analytics.html) provides a way to visualize the flow of events in your OpenSearch data to identify and fix performance problems in distributed applications. - - +Since ingestion and analytics are handled differently in logs and traces, we’ve created separate sections to dive deep into. @@ -90 +80 @@ Security Analytics -Trace Analytics +Logs