AWS opensearch-service documentation change
Summary
Updated documentation for encryption configuration, DLQ setup, and end-to-end acknowledgement. Added UI-based configuration guidance and removed YAML examples.
Security assessment
The changes enhance documentation about security features like customer-managed encryption keys and DLQ IAM permissions, but there is no evidence of addressing a specific security vulnerability. The updates improve clarity around secure configurations without indicating a resolved security flaw.
Diff
diff --git a/opensearch-service/latest/developerguide/osis-features-overview.md b/opensearch-service/latest/developerguide/osis-features-overview.md index 3d1532be9..c593bc57c 100644 --- a//opensearch-service/latest/developerguide/osis-features-overview.md +++ b//opensearch-service/latest/developerguide/osis-features-overview.md @@ -40 +40 @@ A persistent buffer stores your data in a disk-based buffer across multiple Avai -OpenSearch Ingestion dynamically determines the number of OCUs to use for persistent buffering, factoring in the data source, streaming transformations, and sink destination. Because it allocates some OCUs to buffering, you might need to increase the minimum and maximum OCUs to maintain the same ingestion throughput. Pipelines retain data in the buffer for up to 72 hours +OpenSearch Ingestion dynamically determines the number of OCUs to use for persistent buffering, factoring in the data source, streaming transformations, and sink destination. Because it allocates some OCUs to buffering, you might need to increase the minimum and maximum OCUs to maintain the same ingestion throughput. Pipelines retain data in the buffer for up to 72 hours. @@ -70 +70,3 @@ For example: -By default, pipelines use an AWS owned key to encrypt buffer data. These pipelines don't need any additional permissions for the pipeline role. Alternately, you can specify a customer managed key and add the following IAM permissions to the pipeline role: +By default, pipelines use an AWS owned key to encrypt buffer data. These pipelines don't need any additional permissions for the pipeline role. + +Alternately, you can specify a customer managed key and add the following IAM permissions to the pipeline role: @@ -181 +183 @@ A pipeline writes events to the DLQ when either of the following conditions are - * The `max_retries` for the OpenSearch sink have been exhausted. OpenSearch Ingestion requires a minimum of 16 for this option. + * The **Max retries** count for the OpenSearch sink has been exhausted. OpenSearch Ingestion requires a minimum of 16 for this setting. @@ -183 +185 @@ A pipeline writes events to the DLQ when either of the following conditions are - * Events are rejected by the sink due to an error condition. + * The sink is rejecting events due to an error condition. @@ -190 +192 @@ A pipeline writes events to the DLQ when either of the following conditions are -To configure a dead-letter queue for a sub-pipeline, specify the `dlq` option within the `opensearch` sink configuration: +To configure a dead-letter queue for a sub-pipeline, choose **Enable S3 DLQ** when you configure your sink destination. Then, specify the required settings for the queue. For more information, see [Configuration](https://opensearch.org/docs/latest/data-prepper/pipelines/dlq/#configuration) in the Data Prepper DLQ documentation. @@ -192,13 +194 @@ To configure a dead-letter queue for a sub-pipeline, specify the `dlq` option wi - - apache-log-pipeline: - ... - sink: - opensearch: - dlq: - s3: - bucket: "my-dlq-bucket" - key_path_prefix: "dlq-files" - region: "us-west-2" - sts_role_arn: "arn:aws:iam::123456789012:role/dlq-role" - -Files written to this S3 DLQ will have the following naming pattern: +Files written to this S3 DLQ have the following naming pattern: @@ -209,3 +199 @@ Files written to this S3 DLQ will have the following naming pattern: -For more information, see [Dead-Letter Queues (DLQ)](https://opensearch.org/docs/latest/data-prepper/pipelines/dlq/). - -For instructions to configure the `sts_role_arn` role, see [Writing to a dead-letter queue](./pipeline-security-overview.html#pipeline-security-dlq). +For instructions to manually configure the pipeline role to allow access to the S3 bucket that the DLQ writess to, see [Permissions to write to Amazon S3 or a dead-letter queue](./pipeline-security-overview.html#pipeline-security-dlq). @@ -380 +368 @@ For instructions to create index templates that pipelines can use during index c -OpenSearch Ingestion ensures the durability and reliability of data by tracking its delivery from source to sinks in stateless pipelines using _end-to-end acknowledgement_. Currently, only the [S3 source](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/s3/) plugin supports end-to-end acknowledgement. +OpenSearch Ingestion ensures the durability and reliability of data by tracking its delivery from source to sinks in stateless pipelines using _end-to-end acknowledgement_. @@ -382 +370 @@ OpenSearch Ingestion ensures the durability and reliability of data by tracking -With end-to-end acknowledgement, the pipeline source plugin creates an _acknowledgement set_ to monitor a batch of events. It receives a positive acknowledgement when those events are successfully sent to their sinks, or a negative acknowledgement when any of the events could not be sent to their sinks. +###### Note @@ -384 +372 @@ With end-to-end acknowledgement, the pipeline source plugin creates an _acknowle -In the event of a failure or crash of a pipeline component, or if a source fails to receive an acknowledgement, the source times out and takes necessary actions such as retrying or logging the failure. If the pipeline has multiple sinks or multiple sub-pipelines configured, event-level acknowledgements are sent only after the event is sent to _all_ sinks in _all_ sub-pipelines. If a sink has a DLQ configured, end-to-end acknowledgements also tracks events written to the DLQ. +Currently, only the [S3 source](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/s3/) plugin supports end-to-end acknowledgement. @@ -386 +374 @@ In the event of a failure or crash of a pipeline component, or if a source fails -To enable end-to-end acknowledgement, include the `acknowledgments` option within the source configuration: +With end-to-end acknowledgement, the pipeline source plugin creates an _acknowledgement set_ to monitor a batch of events. It receives a positive acknowledgement when those events are successfully sent to their sinks, or a negative acknowledgement when any of the events could not be sent to their sinks. @@ -387,0 +376 @@ To enable end-to-end acknowledgement, include the `acknowledgments` option withi +In the event of a failure or crash of a pipeline component, or if a source fails to receive an acknowledgement, the source times out and takes necessary actions such as retrying or logging the failure. If the pipeline has multiple sinks or multiple sub-pipelines configured, event-level acknowledgements are sent only after the event is sent to _all_ sinks in _all_ sub-pipelines. If a sink has a DLQ configured, end-to-end acknowledgements also tracks events written to the DLQ. @@ -389,5 +378 @@ To enable end-to-end acknowledgement, include the `acknowledgments` option withi - s3-pipeline: - source: - s3: - acknowledgments: true - ... +To enable end-to-end acknowledgement, expand **Additional options** in the Amazon S3 source configuration and choose **Enable end-to-end message acknowledgement**.