AWS opensearch-service documentation change
Summary
Updated IAM role configuration details, added iam:CreateRole and iam:AttachPolicy permissions, restructured pipeline creation steps, and clarified security-related configurations for sinks and network policies.
Security assessment
The changes explicitly document required IAM permissions (iam:CreateRole, iam:AttachPolicy) and emphasize proper pipeline role configuration. This improves clarity around security best practices but does not address a specific disclosed vulnerability. The addition of network policy guidance for Serverless sinks and VPC access configurations further enhances security documentation.
Diff
diff --git a/opensearch-service/latest/developerguide/creating-pipeline.md b/opensearch-service/latest/developerguide/creating-pipeline.md index c3f804a3a..5195c4eb8 100644 --- a//opensearch-service/latest/developerguide/creating-pipeline.md +++ b//opensearch-service/latest/developerguide/creating-pipeline.md @@ -27 +27 @@ For more information, see [Pipelines](https://opensearch.org/docs/latest/clients - * [Using blueprints to create a pipeline](./pipeline-blueprint.html) + * [Working with blueprints](./pipeline-blueprint.html) @@ -36 +36 @@ To create an OpenSearch Ingestion pipeline, you must have the following resource - * An IAM role that OpenSearch Ingestion will assume to write to the sink. You will include this role ARN in your pipeline configuration. + * An IAM role, called the _pipeline role_ , that OpenSearch Ingestion assumes in order to write to the sink. You can create this role ahead of time, or you can have OpenSearch Ingestion create it automatically while you're creating the pipeline. @@ -54 +54 @@ For instructions to create these resources, see the following topics: -If you're writing to a domain that uses fine-grained access control, there are extra steps you need to complete. See [Step 3: Map the pipeline role (only for domains that use fine-grained access control)](./pipeline-domain-access.html#pipeline-access-domain-fgac). +If you're writing to a domain that uses fine-grained access control, there are extra steps you need to complete. See [Map the pipeline role (only for domains that use fine-grained access control)](./pipeline-domain-access.html#pipeline-access-domain-fgac). @@ -64 +64,3 @@ OpenSearch Ingestion uses the following IAM permissions for creating pipelines: - * `iam:PassRole` – Pass the pipeline role to OpenSearch Ingestion so that it can write data to the domain. This permission must be on the [pipeline role resource](./pipeline-domain-access.html#pipeline-access-configure) (the ARN that you specify for the `sts_role_arn` option in the pipeline configuration), or simply `*` if you plan to use different roles in each pipeline. + * `iam:CreateRole` and `iam:AttachPolicy` – Have OpenSearch Ingestion automatically create the pipeline role for you. + + * `iam:PassRole` – Pass the pipeline role to OpenSearch Ingestion so that it can write data to the domain. This permission must be on the [pipeline role resource](./pipeline-domain-access.html#pipeline-access-configure), or simply `*` if you plan to use different roles in each pipeline. @@ -89,0 +92,2 @@ For example, the following policy grants permission to create a pipeline: + "iam:CreateRole", + "iam:AttachPolicy", @@ -106 +110 @@ For more information about each permission, see [Actions, resources, and conditi -When you configure a pipeline, you must specify the major [version of Data Prepper](https://github.com/opensearch-project/data-prepper/releases) that the pipeline will run. To specify the version, include the `version` option in your pipeline configuration: +When you create a pipeline using the configuration editor, you must specify the major [version of Data Prepper](https://github.com/opensearch-project/data-prepper/releases) that the pipeline will run. To specify the version, include the `version` option in your pipeline configuration: @@ -128,2 +132 @@ For pull-based sources like [OTel trace](https://opensearch.org/docs/latest/data -For example, say you specify the following entry sub-pipeline for an ingestion pipeline named `logs`: - +For example, say you specify the following path for a pipeline with an HTTP source: @@ -131,4 +134 @@ For example, say you specify the following entry sub-pipeline for an ingestion p - entry-pipeline: - source: - http: - path: "/my/test_path" + @@ -136 +136 @@ For example, say you specify the following entry sub-pipeline for an ingestion p -When you [ingest data](./configure-client.html) into the pipeline, you must specify the following endpoint in your client configuration: `https://logs-`abcdefgh`.us-west-2.osis.amazonaws.com/`my`/`test_path``. +When you [ingest data](./configure-client.html) into the pipeline, you must specify the following endpoint in your client configuration: `https://`pipeline-name-abc123`.`us-west-2`.osis.amazonaws.com/`my`/`test_path``. @@ -138 +138 @@ When you [ingest data](./configure-client.html) into the pipeline, you must spec -The path must start with a slash (/) and can contain the special characters '-', '_', '.', and '/', as well as the `${pipelineName}` placeholder. If you use `${pipelineName}` (such as `path: "/${pipelineName}/test_path"`), the variable is replaced with the name of the associated sub-pipeline. In this example, it would be `https://logs.us-west-2.osis.amazonaws.com/`entry-pipeline`/`test_path``. +The path must start with a slash (/) and can contain the special characters '-', '_', '.', and '/', as well as the `${pipelineName}` placeholder. If you use `${pipelineName}` (such as `/${pipelineName}/test_path`), OpenSearch Ingestion replaces the variable with the name of the associated sub-pipeline. @@ -144,5 +144 @@ This section describes how to create OpenSearch Ingestion pipelines using the Op -###### To create a pipeline - - 1. Sign in to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home). - - 2. Choose **Pipelines** in the left navigation pane and choose **Create pipeline**. +To create a pipeline, sign in to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home) and choose **Create pipeline**. @@ -150 +146 @@ This section describes how to create OpenSearch Ingestion pipelines using the Op - 3. Either select a blank pipeline, or choose a configuration blueprint. Blueprints include a preconfigured YAML and JSON configuration file for a variety of common use cases. For more information, see [Using blueprints to create a pipeline](./pipeline-blueprint.html). +Either select a blank pipeline, or choose a configuration blueprint. Blueprints include a preconfigured pipeline for a variety of common use cases. For more information, see [Working with blueprints](./pipeline-blueprint.html). @@ -154 +150 @@ Choose **Select blueprint**. - 4. Enter a name for the pipeline. +#### Configure source @@ -156 +152 @@ Choose **Select blueprint**. - 5. (Optional) Choose **Enable persistent buffer**. A persistent buffer stores your data in a disk-based buffer across multiple AZs. For more information, see [Persistent buffering](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-features-overview.html#persistent-buffering). If you enable persistent buffer, select the AWS Key Management Service key to encrypt the buffer data. + 1. If you're starting from a blank pipeline, select a source from the dropdown menu. Available sources might include other AWS services, OpenTelemetry, or HTTP. For more information, see [Integrating Amazon OpenSearch Ingestion pipelines with other services and applications](./configure-client.html). @@ -158,7 +154 @@ Choose **Select blueprint**. - 6. Configure the minimum and maximum pipeline capacity in Ingestion OpenSearch Compute Units (OCUs). For more information, see [Scaling pipelines in Amazon OpenSearch Ingestion](./ingestion-scaling.html). - - 7. Under **Pipeline configuration** , provide your pipeline configuration in YAML format. If you're using a blueprint, the configuration is already pre-populated, but you must make some modifications. - -A single pipeline configuration file can contain 1-10 sub-pipelines. Each sub-pipeline is a combination of a single source, zero or more processors, and a single sink. For OpenSearch Ingestion, the sink must always be an OpenSearch Service domain. For a list of supported options, see [Supported plugins and options for Amazon OpenSearch Ingestion pipelines](./pipeline-config-reference.html). - -###### Note + 2. Depending on which source you choose, configure additional settings for the source. For example, to use Amazon S3 as a source, you must specify the URL of the Amazon SQS queue from the pipeline receives messagess. For a list of supported source plugins and links to their documentation, see [Supported plugins and options for Amazon OpenSearch Ingestion pipelines](./pipeline-config-reference.html). @@ -166,29 +156 @@ A single pipeline configuration file can contain 1-10 sub-pipelines. Each sub-pi -You must include the `sts_role_arn` option in each sub-pipeline. The pipeline assumes the role defined in `sts_role_arn` to sign requests to the domain. For more information, see [Granting Amazon OpenSearch Ingestion pipelines access to domains](./pipeline-domain-access.html). - -The following sample configuration file uses the HTTP source and Grok plugins to process unstructured log data and send it to an OpenSearch Service domain. The sub-pipeline is named `log-pipeline`. - - version: "2" - log-pipeline: - source: - http: - path: "/log/ingest" - processor: - - grok: - match: - log: [ '%{COMMONAPACHELOG}' ] - - date: - from_time_received: true - destination: "@timestamp" - sink: - - opensearch: - hosts: [ "https://search-my-domain.us-east-1.es.amazonaws.com" ] - index: "apache_logs" - aws: - sts_role_arn: "arn:aws:iam::123456789012:role/{pipeline-role}" - region: "us-east-1" - -You can build your own pipeline configuration, or choose **Upload file** and import an existing configuration for a self-managed Data Prepper pipeline. Alternatively, you can use a [configuration blueprint](./pipeline-blueprint.html). - - 8. After you configure your pipeline, choose **Validate pipeline** to confirm that your configuration is correct. If the validation fails, fix the errors and re-run the validation. - - 9. Under **Network configuration** , choose either **VPC access** or **Public access**. If you choose **Public access** , skip to the next step. If you choose **VPC access** , configure the following settings: + 3. For some sources, you must specify **Source network options**. Choose either **VPC access** or **Public access**. If you choose **Public access** , skip to the next step. If you choose **VPC access** , configure the following settings: @@ -199 +161 @@ Setting | Description -**VPC** | Choose the ID of the virtual private cloud VPC that you want to use. The VPC and pipeline must be in the same AWS Region. +**VPC** | Choose the ID of the VPC that you want to use. The VPC and pipeline must be in the same AWS Region. @@ -206 +168,4 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio - 10. (Optional) Under **Tags** , add one or more tags (key-value pairs) to your pipeline. For more information, see [Tagging Amazon OpenSearch Ingestion pipelines](./tag-pipeline.html). + 4. Choose **Next**. + + + @@ -208 +173 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio - 11. (Optional) Under **Log publishing options** , turn on pipeline log publishing to Amazon CloudWatch Logs. We recommend that you enable log publishing so that you can more easily troubleshoot pipeline issues. For more information, see [Monitoring pipeline logs](./monitoring-pipeline-logs.html). +#### Configure processor @@ -210 +175 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio - 12. Choose **Next**. +Add one or more processors to your pipeline. Processors are components within a sub-pipeline that let you filter, transform, and enrich events before publishing records to the domain or collection sink. For a list of supported processors and links to their documentation, see [Supported plugins and options for Amazon OpenSearch Ingestion pipelines](./pipeline-config-reference.html). @@ -212 +177 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio - 13. Review your pipeline configuration and choose **Create**. +You can choose **Actions** and add the following: @@ -213,0 +179 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio + * **Conditional routing** – Routes events to different sinks based on specific conditions. For more information, see [Conditional routing](https://opensearch.org/docs/latest/data-prepper/pipelines/pipelines/#conditional-routing). @@ -214,0 +181,37 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio + * **Sub-pipeline** – Each sub-pipeline is a combination of a single source, zero or more processors, and a single sink. Only one sub-pipeline can have an external source. All others must have sources that are other sub-pipelines within the overall pipeline configuration. A single pipeline configuration can contain 1-10 sub-pipelines. + + + + +Choose **Next**. + +#### Configure sink + +Select the destination where the pipeline publishes records. Every sub-pipeline must contain at least one sink. You can add a maximum of 10 sinks to a pipeline. + +For OpenSearch sinks, configure the following fields: + +Setting | Description +---|--- +**Network policy name**(Serverless sinks only) | If you selected an OpenSearch Serverless collection, enter a **Network policy name**. OpenSearch Ingestion either creates the policy if it doesn't exist, or updates it with a rule that grants access to the VPC endpoint connecting the pipeline and the collection. For more information, see [Granting Amazon OpenSearch Ingestion pipelines access to collections](./pipeline-collection-access.html). +**Index name** | The name of the index where the pipeline sends data. OpenSearch Ingestion creates this index if it doesn't already exist. +**Index mapping options** | Choose how the pipeline stores and indexes documents and their fields into the OpenSearch sink. If you select **Dynamic mapping** , OpenSearch adds fields automatically when you index a document. If you select **Customize mapping** , enter an index mapping template. For more information, see [Index templates](https://opensearch.org/docs/latest/im-plugin/index-templates/). +**Enable DLQ** | Configure an Amazon S3 dead-letter queue (DLQ) for the pipeline. For more information, see [Dead-letter queues](./osis-features-overview.html#osis-features-dlq). +**Additional settings** | Configure advanced options for the OpenSearch sink. For more information, see [Configuration options](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sinks/opensearch/#configuration-options) in the Data Prepper documentation. + +To add an Amazon S3 sink, choose **Add sink** and **Amazon S3**. For more information, see [Amazon S3 as a destination](./configure-client-s3.html#s3-destination). + +Choose **Next**. + +#### Configure pipeline + +Configure the following additional pipeline settings: + +Setting | Description +---|--- +**Pipeline name** | A unique name for the pipeline. +**Persistent buffer** | A persistent buffer stores your data in a disk-based buffer across multiple Availability Zones. For more information, see [Persistent buffering](./osis-features-overview.html#persistent-buffering). If you enable persistent buffering, select the AWS Key Management Service key to encrypt the buffer data. +**Pipeline capacity** | The minimum and maximum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs). For more information, see [Scaling pipelines in Amazon OpenSearch Ingestion](./ingestion-scaling.html). +**Pipeline role** | The IAM role that provides the required permissions for the pipeline to write to the sink and read from pull-based sources. You can create the role yourself, or have OpenSearch Ingestion create it for you based on your selected use case. For more information, see [Setting up roles and users in Amazon OpenSearch Ingestion](./pipeline-security-overview.html). +**Tags** | Add one or more tags to your pipeline. For more information, see [Tagging Amazon OpenSearch Ingestion pipelines](./tag-pipeline.html). +**Log publishing options** | Enable pipeline log publishing to Amazon CloudWatch Logs. We recommend that you enable log publishing so that you can more easily troubleshoot pipeline issues. For more information, see [Monitoring pipeline logs](./monitoring-pipeline-logs.html). @@ -215,0 +219 @@ For more information, see [Configuring VPC access for Amazon OpenSearch Ingestio +Choose **Next**., then review your pipeline configuration and choose **Create pipeline**. @@ -219 +223 @@ OpenSearch Ingestion runs an asynchronous process to build the pipeline. Once th -The [create-pipeline](https://docs.aws.amazon.com/cli/latest/reference/osis/create-pipeline.html) command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with `\n`. For example, `"log-pipeline:\n source:\n http:\n processor:\n - grok:\n ...` +The [create-pipeline](https://docs.aws.amazon.com/cli/latest/reference/osis/create-pipeline.html) command accepts the pipeline configuration as a string or within a .yaml or .json file. If you provide the configuration as a string, each new line must be escaped with `\n`. For example, `"log-pipeline:\n source:\n http:\n processor:\n - grok:\n ...` @@ -239 +243,2 @@ The following sample command creates a pipeline with the following configuration - --pipeline-configuration-body "file://pipeline-config.yaml" + --pipeline-configuration-body "file://pipeline-config.yaml" \ + --pipeline-role-arn arn:aws:iam::1234456789012:role/pipeline-role @@ -303 +308 @@ Pipeline features -Using blueprints to create a pipeline +Working with blueprints