AWS opensearch-service documentation change
Summary
Complete restructuring and rewrite of the 'Logs' documentation section. The content was updated from a general overview of log analysis tools (PPL, AI, SQL, DQL) to a detailed guide focused on the new 'Discover Logs' page, including accessing the page, exploring data, querying with PPL, and creating/adding visualizations to dashboards.
Security assessment
The changes are purely functional and instructional, detailing a new user interface and workflow for log analysis. There is no mention of security vulnerabilities, patches, access controls, encryption, authentication, or any other security-related concepts. The update appears to be a feature enhancement and documentation reorganization.
Diff
diff --git a/opensearch-service/latest/developerguide/observability-analyze-logs.md b/opensearch-service/latest/developerguide/observability-analyze-logs.md index 392e89952..7f13a244b 100644 --- a//opensearch-service/latest/developerguide/observability-analyze-logs.md +++ b//opensearch-service/latest/developerguide/observability-analyze-logs.md @@ -5 +5 @@ -OpenSearch UI and observability workspaceQuerying your logs using PPLQuerying your logs using AIQuerying your logs using SQLQuerying your logs using DQLDashboards and alerts for logs +To access the Logs pageExploring log dataQuerying logs using PPLCreating visualizations from logsAdding visualizations to dashboards @@ -7 +7 @@ OpenSearch UI and observability workspaceQuerying your logs using PPLQuerying yo -# Logs +# Discover Logs @@ -9 +9 @@ OpenSearch UI and observability workspaceQuerying your logs using PPLQuerying yo -OpenSearch Ingestion can transform unstructured log data into a structured format during ingestion. OpenSearch Ingestion provides processors that normalize and enrich your data before it is indexed. Examples of helpful processors are: +The Discover Logs page provides a dedicated interface for exploring and analyzing log data in your OpenSearch Service observability workspace. You can write PPL queries to filter and aggregate log data, create visualizations directly from query results, and add those visualizations to dashboards. The page also provides natural language query assistance powered by the OpenSearch AI assistant. @@ -11 +11 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma - * `grok` – Parses and structures unstructured text data such as web server access logs, into distinct fields. +## To access the Logs page @@ -13 +13 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma - * `date` – Parses a date from a log field and sets it as the event's timestamp. +In your observability workspace, expand **Discover** in the left navigation and choose **Logs**. @@ -15 +15 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma - * `parse_json` – Parses a string field that contains a JSON object. +## Exploring log data @@ -16,0 +17 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma +The Discover Logs interface provides the following components for exploring your log data. @@ -17,0 +19 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma + @@ -18,0 +21 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma + * **Dataset selector** – Choose the logs dataset that you want to query. Each dataset maps to one or more indexes in your OpenSearch Service domain. @@ -20 +23 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma -**Note** – To make getting started easier, we’ve created a new [Get Started](https://us-east-1.console.aws.amazon.com/aos/home#/opensearch/getting-started) workflow for logs in the Amazon OpenSearch Service console which will set up a new Otel tailored ingestion pipeline, point it to an existing OpenSearch cluster, and create a new OpenSearch UI application with an observability workspace created. All you have to do is point your Otel agents to the new ingestion endpoint. + * **Query editor** – Write PPL queries to filter, aggregate, and transform your log data. The editor provides autocomplete suggestions and syntax highlighting. @@ -22 +25 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma -## OpenSearch UI and observability workspace + * **Time filter** – Specify the time range for your query results. You can choose a relative range or specify absolute start and end times. @@ -24 +27 @@ OpenSearch Ingestion can transform unstructured log data into a structured forma -After your logs data is ingested into Amazon OpenSearch Service, you use the tools provided by the Amazon OpenSearch Service observability workspace in OpenSearch UI to analyze it. The observability workspace provides specialized tools designed to extract meaningful insights in Discover and Dashboards. + * **Results panel** – View query results as a table of log events. You can expand individual events to see all fields. @@ -26 +29 @@ After your logs data is ingested into Amazon OpenSearch Service, you use the too -The observability workspace comes with a new Discover experience which uses [piped processing language](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/) (PPL) complemented with a natural language assistant powered by Amazon Q Developer for Business. The language assistance makes it simple for anyone to get started with piped languages. After refining your query, create visualizations and dashboards right from new Discover without jumping to other parts of the tool. To query your data using [DQL](https://docs.opensearch.org/latest/dashboards/dql/) or [SQL](https://github.com/opensearch-project/sql/blob/main/docs/user/index.rst), switch to the old Discover experience. + * **Histogram** – View the distribution of log events over time. The histogram updates automatically based on your query and time filter. @@ -28 +31 @@ The observability workspace comes with a new Discover experience which uses [pip - + * **Fields panel** – Browse available fields in your dataset and add them as columns to the results table. @@ -30 +32,0 @@ The observability workspace comes with a new Discover experience which uses [pip -## Querying your logs using PPL @@ -32 +33,0 @@ The observability workspace comes with a new Discover experience which uses [pip -You have several options for querying your logs to gather insights into the operation of your application or service. @@ -34 +34,0 @@ You have several options for querying your logs to gather insights into the oper -Piped processing language (PPL) is a query language with pipe-based (|) syntax for chaining commands. You can use it to build powerful expressions to analyze your logs. @@ -36 +36 @@ Piped processing language (PPL) is a query language with pipe-based (|) syntax f -**Note** : To unlock newer PPL commands/functions in OpenSearch 2.19, you’ll need to change a feature flag in OpenSearch Developer Tools using the following query (not required for OpenSearch 3.3): +## Querying logs using PPL @@ -37,0 +38 @@ Piped processing language (PPL) is a query language with pipe-based (|) syntax f +Piped processing language (PPL) is a query language that uses pipe-based (`|`) syntax for chaining commands. You can use PPL to filter, aggregate, and transform your log data. @@ -39 +40 @@ Piped processing language (PPL) is a query language with pipe-based (|) syntax f - PUT /_plugins/_query/settings { "transient" : { "plugins.calcite.enabled" : true } } +### Basic queries @@ -41 +42 @@ Piped processing language (PPL) is a query language with pipe-based (|) syntax f -### Find the hosts with the most errors +To retrieve all log events from a dataset, use the `source` command: @@ -43 +43,0 @@ Piped processing language (PPL) is a query language with pipe-based (|) syntax f -This example analyzes you logs to determine the service hosts with the most total errors. @@ -44,0 +45 @@ This example analyzes you logs to determine the service hosts with the most tota + source = my-logs-dataset @@ -46,5 +47 @@ This example analyzes you logs to determine the service hosts with the most tota - source = my-index | - where level = "ERROR" | - stats count() as error_count by host | - sort -error_count | - head 5 +To limit the number of results, use the `head` command: @@ -52 +48,0 @@ This example analyzes you logs to determine the service hosts with the most tota -### Calculate average request time @@ -54 +50 @@ This example analyzes you logs to determine the service hosts with the most tota -This example analyzes your logs to calculate the average request time for each status code in the log. + source = my-logs-dataset | head 20 @@ -55,0 +52 @@ This example analyzes your logs to calculate the average request time for each s +### Filtering with WHERE @@ -57,2 +54 @@ This example analyzes your logs to calculate the average request time for each s - source = my-index | - stats avg(request_time) by status_code +Use the `where` clause to filter log events based on field values: @@ -60 +55,0 @@ This example analyzes your logs to calculate the average request time for each s -For more information about PPL, see the [PPL reference manual](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/) on opensearch.org. @@ -62 +57 @@ For more information about PPL, see the [PPL reference manual](https://docs.open -## Querying your logs using AI + source = my-logs-dataset | where severity_text = 'ERROR' @@ -64 +59 @@ For more information about PPL, see the [PPL reference manual](https://docs.open -This example analyzes your logs to show the errors logged in the last 5 minutes. +You can combine multiple conditions: @@ -67 +62,2 @@ This example analyzes your logs to show the errors logged in the last 5 minutes. - Show me all of the error logs from the last 5 minutes + source = my-logs-dataset | + where severity_text = 'ERROR' and service_name = 'payment-service' @@ -69 +65 @@ This example analyzes your logs to show the errors logged in the last 5 minutes. - +### Managing queries @@ -71 +67 @@ This example analyzes your logs to show the errors logged in the last 5 minutes. -## Querying your logs using SQL +You can save frequently used queries for reuse. To save a query, choose **Save** in the query editor toolbar and enter a name for the query. To load a saved query, choose **Open** and select the query from the list. @@ -73 +69 @@ This example analyzes your logs to show the errors logged in the last 5 minutes. -SQL provides a familiar way to query log data. +For the complete list of PPL commands and functions, see the [Piped Processing Language reference](https://observability.opensearch.org/docs/ppl/). @@ -75 +71 @@ SQL provides a familiar way to query log data. -This example analyzes your logs to show errors by timestamp. +## Creating visualizations from logs @@ -76,0 +73 @@ This example analyzes your logs to show errors by timestamp. +You can create visualizations directly from your PPL query results. Use the `stats` command to aggregate data for visualization: @@ -78,4 +74,0 @@ This example analyzes your logs to show errors by timestamp. - SELECT timestamp, severity_text, body, service_name - FROM opentelemetry_logs - WHERE severity_text = 'ERROR' AND service_name = 'my-service' - ORDER BY timestamp DESC; @@ -83 +76,2 @@ This example analyzes your logs to show errors by timestamp. -For more information about SQL, see the [SQL reference manual](https://github.com/opensearch-project/sql/blob/main/docs/user/index.rst) on GitHub. + source = my-logs-dataset | + stats count() as error_count by service_name, span(timestamp, 1h) @@ -85 +79 @@ For more information about SQL, see the [SQL reference manual](https://github.co -## Querying your logs using DQL +After you run a `stats` query, choose the **Visualization** tab to see the results as a chart. @@ -87 +81 @@ For more information about SQL, see the [SQL reference manual](https://github.co -DQL is good for quick searching and filtering. + @@ -89 +83 @@ DQL is good for quick searching and filtering. -This example analyzes your logs and returns errors and exceptions. +### Visualization types @@ -90,0 +85 @@ This example analyzes your logs and returns errors and exceptions. +The following table describes the visualization types that you can use. @@ -92 +87,10 @@ This example analyzes your logs and returns errors and exceptions. - error OR exception +Type | Description +---|--- +Line | Displays data points connected by lines, useful for showing trends over time. +Area | Similar to a line chart with the area under the line filled in, useful for showing volume over time. +Bar | Displays data as vertical or horizontal bars, useful for comparing values across categories. +Metric | Displays a single numeric value, useful for showing key performance indicators. +State timeline | Displays state changes over time as colored bands, useful for monitoring status transitions. +Heatmap | Displays data as a matrix of colored cells, useful for showing density and patterns. +Bar gauge | Displays a single value as a filled bar within a range, useful for showing progress toward a threshold. +Pie | Displays data as proportional slices of a circle, useful for showing composition. @@ -94 +98 @@ This example analyzes your logs and returns errors and exceptions. -For more information about DQL, see the [DQL reference manual](https://docs.opensearch.org/latest/dashboards/dql/) on opensearch.org. + @@ -96 +100 @@ For more information about DQL, see the [DQL reference manual](https://docs.open -## Dashboards and alerts for logs +### Visualization settings @@ -98 +102 @@ For more information about DQL, see the [DQL reference manual](https://docs.open -In the new Discover experience with PPL, you can create visualizations from the visualizations tab within Discover. Choose from 12 visualization types and edit on the fly before adding them to a dashboard. In the old Discover experience, you’ll browse to Visualize in the left navigation to create a new visualization and to Dashboards to add the visualizations to your dashboards. +When the **Visualization** tab is active, a settings panel appears on the right side of the screen. Use this panel to configure the chart type, map fields to axes, and customize visual styles such as colors and legends. @@ -100 +104 @@ In the new Discover experience with PPL, you can create visualizations from the - +To switch the axes of a visualization, use the axis configuration in the settings panel. @@ -102 +106 @@ In the new Discover experience with PPL, you can create visualizations from the -You can define alert monitors using PPL or the OpenSearch Service query DSL to run scheduled queries. A trigger condition, such as a specific number of error logs, fires an alert. You can send notifications through channels such Amazon Simple Notification Service or webhooks. + @@ -104 +108,3 @@ You can define alert monitors using PPL or the OpenSearch Service query DSL to r -For more information about alerting, see the [alerting documentation](https://docs.opensearch.org/latest/observing-your-data/alerting/index/) on opensearch.org. +## Adding visualizations to dashboards + +After you create a visualization, you can add it to a dashboard for ongoing monitoring. Choose **Save to dashboard** in the visualization toolbar, then select an existing dashboard or create a new one. The visualization is saved with its underlying PPL query so that it refreshes automatically when you open the dashboard. @@ -112 +118 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Observability +Datasets @@ -114 +120 @@ Observability -Traces +Discover traces