AWS opensearch-service documentation change
Summary
Comprehensive rewrite of OpenTelemetry configuration documentation, adding sections for prerequisites, step-by-step setup, cross-account connectivity, limitations, and recommended CloudWatch alarms. Expanded coverage to include logs and metrics ingestion alongside traces.
Security assessment
Added security documentation about required IAM permissions ('osis:Ingest') for ingestion roles and cross-account pipeline security considerations. However, there's no evidence of addressing a specific vulnerability or incident. The changes primarily improve operational documentation.
Diff
diff --git a/opensearch-service/latest/developerguide/configure-client-otel.md b/opensearch-service/latest/developerguide/configure-client-otel.md index 8cdb0708c..830dda8c0 100644 --- a//opensearch-service/latest/developerguide/configure-client-otel.md +++ b//opensearch-service/latest/developerguide/configure-client-otel.md @@ -4,0 +5,2 @@ +PrerequisitesStep 1: Configure the pipeline roleStep 2: Create the pipelineCross-account ConnectivityLimitationsRecommended CloudWatch Alarms for OpenTelemetry sources + @@ -7 +9,13 @@ -This sample [OpenTelemetry configuration file](https://opentelemetry.io/docs/collector/configuration/) exports trace data from the OpenTelemetry Collector and sends it to an OpenSearch Ingestion pipeline. For more information about ingesting trace data, see [Trace Analytics](https://opensearch.org/docs/latest/data-prepper/common-use-cases/trace-analytics/) in the Data Prepper documentation. +You can use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to ingest logs, traces, and metrics into OpenSearch Ingestion pipelines. A single pipeline can be used to ingest all logs, traces, and metrics to different indices on a domain or collection. You can also use pipelines to ingest only logs, traces, or metrics individually. + +###### Topics + + * Prerequisites + + * Step 1: Configure the pipeline role + + * Step 2: Create the pipeline + + * Cross-account Connectivity + + * Limitations @@ -9 +23 @@ This sample [OpenTelemetry configuration file](https://opentelemetry.io/docs/col -Note the following: + * Recommended CloudWatch Alarms for OpenTelemetry sources @@ -11 +24,0 @@ Note the following: - * The `endpoint` value must include your pipeline endpoint. For example, `https://`pipeline-endpoint`.`us-east-1`osis.amazonaws.com`. @@ -13 +25,0 @@ Note the following: - * The `service` value must be `osis`. @@ -15 +27,12 @@ Note the following: - * The `compression` option for the OTLP/HTTP Exporter must match the `compression` option on the pipeline's OpenTelemetry source. + +## Prerequisites + +While setting up the [OpenTelemetry configuration file](https://opentelemetry.io/docs/collector/configuration/), you must configure the following in order for ingestion to occur: + + * The ingestion role needs the `osis:Ingest` permission to interact with the pipeline. For more information, see [Ingestion role](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-security-overview.html#pipeline-security-same-account). + + * The endpoint value must include your pipeline endpoint. For example, `https://pipeline-endpoint.us-east-1.osis.amazonaws.com.` + + * The service value must be `osis`. + + * The compression option for the OTLP/HTTP Exporter must match the compression option on the pipeline's selected source. @@ -24,6 +47 @@ Note the following: - service: "**osis** " - - receivers: - jaeger: - protocols: - grpc: + service: "osis" @@ -32,0 +51,2 @@ Note the following: + logs_endpoint: "https://pipeline-endpoint.us-east-1.osis.amazonaws.com/v1/logs" + metrics_endpoint: "https://pipeline-endpoint.us-east-1.osis.amazonaws.com/v1/metrics" @@ -45 +64,0 @@ Note the following: -You can then configure an OpenSearch Ingestion pipeline like the following, which specifies the [OTel trace](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/otel-trace/) plugin as the source: @@ -46,0 +66 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic +## Step 1: Configure the pipeline role @@ -48,2 +68,11 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - version: "2" - otel-trace-pipeline: +After setting up the OpenTelemetry collector configuration, [ set up the pipeline role](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-security-overview.html#pipeline-security-sink) that you want to use in your pipeline configuration. There are not specific permissions that the pipeline role needs for the OTLP source, only permissions to grant pipelines access to the OpenSearch domain or collection. + +## Step 2: Create the pipeline + +You can then configure an OpenSearch Ingestion pipeline like the following, which specifies OTLP as the source. You can also configure OpenTelemetry logs, metrics, and traces as individual sources. + +OTLP source pipeline configuration: + + + version: 2 + otlp-pipeline: @@ -51,4 +80,21 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - otel_trace_source: - path: "/v1/traces" - processor: - - trace_peer_forwarder: + otlp: + logs_path: /otlp-pipeline/v1/logs + traces_path: /otlp-pipeline/v1/traces + metrics_path: /otlp-pipeline/v1/metrics + sink: + - opensearch: + hosts: ["https://search-mydomain.region.es.amazonaws.com"] + index: "ss4o_metrics-otel-%{yyyy.MM.dd}" + index_type: custom + aws: + region: "region" + + +OpenTelemetry Logs pipeline configuration: + + + version: 2 + otel-logs-pipeline: + source: + otel_logs_source: + path: /otel-logs-pipeline/v1/logs @@ -56,5 +102,13 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - - pipeline: - name: "trace-pipeline" - - pipeline: - name: "service-map-pipeline" - trace-pipeline: + - opensearch: + hosts: ["https://search-mydomain.region.es.amazonaws.com"] + index: "ss4o_metrics-otel-%{yyyy.MM.dd}" + index_type: custom + aws: + region: "region" + + +OpenTelemetry Metrics pipeline configuration: + + + version: 2 + otel-metrics-pipeline: @@ -62,4 +116,2 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - pipeline: - name: "otel-trace-pipeline" - processor: - - otel_traces: + otel_metrics_source: + path: /otel-metrics-pipeline/v1/metrics @@ -68,2 +120,3 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - hosts: ["https://search-domain-endpoint.us-east-1.es.amazonaws.com"] - index_type: trace-analytics-raw + hosts: ["https://search-mydomain.region.es.amazonaws.com"] + index: "ss4o_metrics-otel-%{yyyy.MM.dd}" + index_type: custom @@ -72 +125,7 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - service-map-pipeline: + + +OpenTelemetry Traces pipeline configuration: + + + version: 2 + otel-trace-pipeline: @@ -74,4 +133,2 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - pipeline: - name: "otel-trace-pipeline" - processor: - - service_map: + otel_trace_source: + path: /otel-traces-pipeline/v1/traces @@ -80,2 +137,3 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - hosts: ["https://search-domain-endpoint.us-east-1.es.amazonaws.com"] - index_type: trace-analytics-service-map + hosts: ["https://search-mydomain.region.es.amazonaws.com"] + index: "ss4o_metrics-otel-%{yyyy.MM.dd}" + index_type: custom @@ -85 +143,48 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic -For another example pipeline, see the preconfigured trace analytics blueprint. For more information, see [Working with blueprints](./pipeline-blueprint.html). + +You can use a preconfigured blueprint to create this pipeline. For more information, see [Working with blueprints](./pipeline-blueprint.html). + +## Cross-account Connectivity + +OpenSearch Ingestion pipelines with OpenTelemetry sources have cross-account ingestion capability. Amazon OpenSearch Ingestion enables you to share pipelines across AWS accounts from a virtual private cloud (VPC) to a pipeline endpoint in a separate VPC. For more information, see [Configuring OpenSearch Ingestion pipelines for cross-account ingestion](./cross-account-pipelines.html). + +## Limitations + +The OpenSearch Ingestion pipeline cannot receive any requests greater than 20mb. This value is configured by the user in the `max_request_length` option. This option defaults to 10mb. + +## Recommended CloudWatch Alarms for OpenTelemetry sources + +The following CloudWatch metrics are recommended for monitoring the performance of your ingestion pipeline. These metrics can help you identify the amount of data processed from exports, the amount of events processed from streams, the errors in processing exports and stream events, and the number of documents written to the destination. You can setup CloudWatch alarms to perform an action when one of these metrics exceed a specified value for a specified amount of time. + +The CloudWatch metrics for OTLP source are formatted as `{pipeline-name}.otlp.{logs | traces | metrics}.{metric-name}`. For example, `otel-pipeline.otlp.metrics.requestTimeouts.count`. + +In the case of using an individual OpenTelemetry source, the metrics will be formatted as `{pipeline-name}.{source-name}.{metric-name}`. For example, `trace-pipeline.otel_trace_source.requestTimeouts.count`. + +All three OpenTelemetry data types will have the same metrics, but for brevity the metrics will only be listed in the below table for OTLP source log type data. + +Metric | Description +---|--- +`otel-pipeline.BlockingBuffer.bufferUsage.value` | Indicates how much of the buffer is being utilized. +`otel-pipeline.otlp.logs.requestTimeouts.count` | The number of requests that have timed out. +`otel-pipeline.otlp.logs.requestsReceived.count` | The number of requests received by the OpenTelemetry Collector. +`otel-pipeline.otlp.logs.badRequests.count` | The number of malformed requests received by the OpenTelemetry Collector. +`otel-pipeline.otlp.logs.requestsTooLarge.count` | The number of requests greater than the maximum of 20mb received by the OpenTelemetry Collector. +`otel-pipeline.otlp.logs.internalServerError.count` | The number of HTTP 500 errors received from the OpenTelemetry Collector. +`otel-pipeline.opensearch.bulkBadRequestErrors.count` | Count of errors during bulk requests due to malformed request. +`otel-pipeline.opensearch.bulkRequestLatency.avg` | Average latency for bulk write requests made to OpenSearch. +`otel-pipeline.opensearch.bulkRequestNotFoundErrors.count` | Number of bulk requests that failed because the target data could not be found. +`otel-pipeline.opensearch.bulkRequestNumberOfRetries.count` | Number of retries by OpenSearch Ingestion pipelines to write OpenSearch cluster. +`otel-pipeline.opensearch.bulkRequestSizeBytes.sum` | Total size in bytes of all bulk requests made to OpenSearch. +`otel-pipeline.opensearch.documentErrors.count` | Number of errors when sending documents to OpenSearch. The documents causing the errors witll be sent to DLQ. +`otel-pipeline.opensearch.documentsSuccess.count` | Number of documents successfully written to an OpenSearch cluster or collection. +`otel-pipeline.opensearch.documentsSuccessFirstAttempt.count` | Number of documents successfully indexed in OpenSearch on the first attempt. +`otel-pipeline.opensearch.documentsVersionConflictErrors.count` | Count of errors due to version conflicts in documents during processing. +`otel-pipeline.opensearch.PipelineLatency.avg` | Average latency of OpenSearch Ingestion pipeline to process the data by reading from the source to writing to the destination. +`otel-pipeline.opensearch.PipelineLatency.max` | Maximum latency of OpenSearch Ingestion pipeline to process the data by reading from the source to writing the destination. +`otel-pipeline.opensearch.recordsIn.count` | Count of records successfully ingested into OpenSearch. This metric is essential for tracking the volume of data being processed and stored. +`otel-pipeline.opensearch.s3.dlqS3RecordsFailed.count` | Number of records that failed to write to DLQ. +`otel-pipeline.opensearch.s3.dlqS3RecordsSuccess.count` | Number of records that are written to DLQ. +`otel-pipeline.opensearch.s3.dlqS3RequestLatency.count` | Count of latency measurements for requests to the Amazon S3 dead-letter queue. +`otel-pipeline.opensearch.s3.dlqS3RequestLatency.sum` | Total latency for all requests to the Amazon S3 dead-letter queue +`otel-pipeline.opensearch.s3.dlqS3RequestSizeBytes.sum` | Total size in bytes of all requests made to the Amazon S3 dead-letter queue.