AWS opensearch-service high security documentation change
Summary
Restructured IAM role documentation, removed management role details, added automated pipeline role creation, expanded pipeline/ingestion role guidance, and added cross-account ingestion instructions.
Security assessment
The change introduces automated pipeline role creation to reduce configuration errors and enforce security best practices, explicitly stating it 'enhances security compliance'. It also adds detailed guidance for cross-account ingestion and secure permissions configurations, directly improving security documentation.
Diff
diff --git a/opensearch-service/latest/developerguide/pipeline-security-overview.md b/opensearch-service/latest/developerguide/pipeline-security-overview.md index 5a37c6958..9f6c53b03 100644 --- a//opensearch-service/latest/developerguide/pipeline-security-overview.md +++ b//opensearch-service/latest/developerguide/pipeline-security-overview.md @@ -5 +5 @@ -Management rolePipeline roleIngestion role +Pipeline roleIngestion role @@ -9 +9 @@ Management rolePipeline roleIngestion role -Amazon OpenSearch Ingestion uses a variety of permissions models and IAM roles to allow source applications to write to pipelines, and to allow pipelines to write to sinks. Before you can start ingesting data, you must create one or more IAM roles with specific permissions based on your use case. +Amazon OpenSearch Ingestion uses a variety of permissions models and IAM roles in order to allow source applications to write to pipelines, and to allow pipelines to write to sinks. Before you can start ingesting data, you need to create one or more IAM roles with specific permissions based on your use case. @@ -11 +11 @@ Amazon OpenSearch Ingestion uses a variety of permissions models and IAM roles t -At minimum, you require the following roles to set up a successful pipeline. +At minimum, the following roles are required to set up a successful pipeline. @@ -15,7 +15 @@ Name | Description -Management role | Any principal that's managing pipelines (generally a "pipeline admin") needs management access, which includes permissions like `osis:CreatePipeline` and `osis:UpdatePipeline`. These permissions allow a user to administer pipelines but not necessarily write data to them. -Pipeline role | The pipeline role, which you specify within the pipeline's YAML configuration, provides the required permissions for a pipeline to write to the domain or collection sink and read from pull-based sources. For more information, see the following topics: - - * [Granting Amazon OpenSearch Ingestion pipelines access to domains](./pipeline-domain-access.html) - * [Granting Amazon OpenSearch Ingestion pipelines access to collections](./pipeline-collection-access.html) - - +Pipeline role | The pipeline role provides the required permissions for a pipeline to read from the source and write to the domain or collection sink. You can manually create the pipeline role, or you can have OpenSearch Ingestion create it for you. @@ -30 +24 @@ For a simple setup guide, see [Tutorial: Ingesting data into a domain using Amaz -## Management role +**Topics** @@ -32 +26 @@ For a simple setup guide, see [Tutorial: Ingesting data into a domain using Amaz -In addition to the basic `osis:*` permissions needed to create and modify a pipeline, you also need the `iam:PassRole` permission for the pipeline role resource. Any AWS service that accepts a role must use this permission. OpenSearch Ingestion assumes the role every time it needs to write data to a sink. This helps administrators ensure that only approved users can configure OpenSearch Ingestion with a role that grants permissions. For more information, see [Granting a user permissions to pass a role to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html). + * Pipeline role @@ -34 +28 @@ In addition to the basic `osis:*` permissions needed to create and modify a pipe -If you're using the AWS Management Console (using blueprints and later checking on your pipeline), you need the following permissions to create and update a pipeline: + * Ingestion role @@ -35,0 +30 @@ If you're using the AWS Management Console (using blueprints and later checking + * Cross-account ingestion @@ -37,30 +31,0 @@ If you're using the AWS Management Console (using blueprints and later checking - { - "Version":"2012-10-17", - "Statement":[ - { - "Effect":"Allow", - "Resource":"*", - "Action":[ - "osis:CreatePipeline", - "osis:GetPipelineBlueprint", - "osis:ListPipelineBlueprints", - "osis:GetPipeline", - "osis:ListPipelines", - "osis:GetPipelineChangeProgress", - "osis:ValidatePipeline", - "osis:UpdatePipeline" - ] - }, - { - "Resource":[ - "arn:aws:iam::account-id:role/pipeline-role" - ], - "Effect":"Allow", - "Action":[ - "iam:PassRole" - ] - } - ] - } - -If you're using the AWS CLI (not prevalidating your pipeline or using blueprints), you need the following permissions to create and update a pipeline: @@ -69,22 +33,0 @@ If you're using the AWS CLI (not prevalidating your pipeline or using blueprints - { - "Version":"2012-10-17", - "Statement":[ - { - "Effect":"Allow", - "Resource":"*", - "Action":[ - "osis:CreatePipeline", - "osis:UpdatePipeline" - ] - }, - { - "Resource":[ - "arn:aws:iam::account-id:role/pipeline-role" - ], - "Effect":"Allow", - "Action":[ - "iam:PassRole" - ] - } - ] - } @@ -94 +37 @@ If you're using the AWS CLI (not prevalidating your pipeline or using blueprints -A pipeline needs certain permissions to write to its sink. These permissions depend on whether the sink is an OpenSearch Service domain or an OpenSearch Serverless collection. +A pipeline needs certain permissions to read from its source and write to its sink. These permissions depend on the client application or AWS service that is writing to the pipeline, and whether the sink is an OpenSearch Service domain, an OpenSearch Serverless collection, or Amazon S3. In addition, a pipeline might need permissions to physically _pull_ data from the source application (if the source is a pull-based plugin), and permissions to write to an S3 dead letter queue, if enabled. @@ -96 +39,3 @@ A pipeline needs certain permissions to write to its sink. These permissions dep -In addition, a pipeline might need permissions to pull from the source application (if the source is a pull-based plugin), and permissions to write to an S3 dead letter queue, if configured. +When you create a pipeline, you have the option of specifying an existing IAM role that you manually created, or having OpenSearch Ingestion automatically create the pipeline role based on the source and the sink that you selected. The following image shows how to specify the pipeline role in the AWS Management Console. + + @@ -100 +45 @@ In addition, a pipeline might need permissions to pull from the source applicati - * Writing to a domain sink + * Automating pipeline role creation @@ -102 +47 @@ In addition, a pipeline might need permissions to pull from the source applicati - * Writing to a collection sink + * Manually creating the pipeline role @@ -104 +48,0 @@ In addition, a pipeline might need permissions to pull from the source applicati - * Writing to a dead-letter queue @@ -107,0 +52 @@ In addition, a pipeline might need permissions to pull from the source applicati +### Automating pipeline role creation @@ -109 +54 @@ In addition, a pipeline might need permissions to pull from the source applicati -### Writing to a domain sink +You can choose to have OpenSearch Ingestion create the pipeline role for you. It automatically identifies which permissions the role requires based on the configured source and sinks. It creates an IAM role with the prefix `OpenSearchIngestion-`, and with the suffix that you enter. For example, if you enter `PipelineRole` as the suffix, OpenSearch Ingestion creates a role named `OpenSearchIngestion-PipelineRole`. @@ -111 +56 @@ In addition, a pipeline might need permissions to pull from the source applicati -An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Service domain that is configured as its sink. These permissions include the ability to describe the domain and send HTTP requests to it. +Automatically creating the pipeline role simplifies the setup process and reduces the likelihood of configuration errors. By automating the role creation, you can avoid manually assigning permissions, ensuring that the correct policies are applied without risking security misconfigurations. This also saves time and enhances security compliance by enforcing best practices, while ensuring consistency across multiple pipeline deployments. @@ -113 +58 @@ An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Serv -To provide your pipeline with the required permissions to write to a sink, first create an AWS Identity and Access Management (IAM) role with the [required permissions](./pipeline-domain-access.html#pipeline-access-configure). These permissions are the same for public and VPC pipelines. Then, specify the pipeline role in the domain access policy so that the domain can accept write requests from the pipeline. +You can only have OpenSearch Ingestion automatically create the pipeline role in the AWS Management Console. If you're using the AWS CLI, the OpenSearch Ingestion API, or one of the SDKs, you must specify a manually-created pipeline role. @@ -115 +60 @@ To provide your pipeline with the required permissions to write to a sink, first -Finally, specify the role ARN as the value of the **sts_role_arn** option within the pipeline configuration: +To have OpenSearch Ingestion create the role for you, select **Create and use a new service role**. @@ -116,0 +62 @@ Finally, specify the role ARN as the value of the **sts_role_arn** option within +###### Important @@ -118,11 +64 @@ Finally, specify the role ARN as the value of the **sts_role_arn** option within - version: "2" - source: - http: - ... - processor: - ... - sink: - - opensearch: - ... - aws: - **sts_role_arn** : arn:aws:iam::account-id:role/pipeline-role +You still need to manually modify the domain or collection access policy to grant access to the pipeline role. For domains that use fine-grained access control, you must also map the pipeline role to a backend role. You can perform these steps before or after you create the pipeline. @@ -130 +66 @@ Finally, specify the role ARN as the value of the **sts_role_arn** option within -For instructions to complete each of these steps, see [Allowing pipelines to access domains](./pipeline-domain-access.html). +For instructions, see the following topics: @@ -132 +68 @@ For instructions to complete each of these steps, see [Allowing pipelines to acc -### Writing to a collection sink + * [Configure data access for the domain](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-domain-access.html#pipeline-access-domain) @@ -134 +70,20 @@ For instructions to complete each of these steps, see [Allowing pipelines to acc -An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Serverless collection that is configured as its sink. These permissions include the ability to describe the collection and send HTTP requests to it. + * [Configure data and network access for the collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-domain-access.html#pipeline-collection-acces) + + + + +### Manually creating the pipeline role + +You might prefer to manually create the pipeline role if you need more control over permissions to meet specific security or compliance requirements. Manual creation allows you to tailor roles to fit existing infrastructure or access management strategies. You might also choose manual setup to integrate the role with other AWS services or ensure it aligns with your unique operational needs. + +To choose a manually-created pipeline role, select **Use an existing IAM role** and choose an existing role. The role must have all permissions needed to receive data from the selected source and write to the selected sink. The following sections outline how to manually create a pipeline role. + +###### Topics + + * Permissions to read from a source + + * Permissions to write to a domain sink + + * Permissions to write to a collection sink + + * Permissions to write to Amazon S3 or a dead-letter queue @@ -136 +90,0 @@ An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Serv -First, create an IAM role that has the `aoss:BatchGetCollection` permission against all resources (`*`). Then, include this role in a data access policy and provide it permissions to create indexes, update indexes, describe indexes, and write documents within the collection. Finally, specify the role ARN as the value of the **sts_role_arn** option within the pipeline configuration. @@ -138 +91,0 @@ First, create an IAM role that has the `aoss:BatchGetCollection` permission agai -For instructions to complete each of these steps, see [Allowing pipelines to access collections](./pipeline-collection-access.html). @@ -140 +92,0 @@ For instructions to complete each of these steps, see [Allowing pipelines to acc -### Writing to a dead-letter queue @@ -142 +94 @@ For instructions to complete each of these steps, see [Allowing pipelines to acc -If you configure your pipeline to write to a [dead-letter queue](https://opensearch.org/docs/latest/data-prepper/pipelines/dlq/) (DLQ), you must include the `sts_role_arn` option within the DLQ configuration. The permissions included in this role allow the pipeline to access the S3 bucket that you specify as the destination for DLQ events. +#### Permissions to read from a source @@ -144 +96,17 @@ If you configure your pipeline to write to a [dead-letter queue](https://opensea -You must use the same `sts_role_arn` in all pipeline components. Therefore, you must attach a separate permissions policy to your pipeline role that provides DLQ access. At minimum, the role must be allowed the `S3:PutObject` action on the bucket resource: +An OpenSearch Ingestion pipeline needs permission to read and receive data from the specified source. For example, for an Amazon DynamoDB source, it needs permissions such as `dynamodb:DescribeTable` and `dynamodb:DescribeStream`. For sample pipeline role access policies for common sources, such as Amazon S3, Fluent Bit, and the OpenTelemetry Collector, see [Integrating Amazon OpenSearch Ingestion pipelines with other services and applications](./configure-client.html). + +#### Permissions to write to a domain sink + +An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Service domain that is configured as its sink. These permissions include the ability to describe the domain and send HTTP requests to it. These permissions are the same for public and VPC domains. For instructions to create a pipeline role and specify it in the domain access policy, see [Allowing pipelines to access domains](./pipeline-domain-access.html). + +#### Permissions to write to a collection sink + +An OpenSearch Ingestion pipeline needs permission to write to an OpenSearch Serverless collection that is configured as its sink. These permissions include the ability to describe the collection and send HTTP requests to it. + +First, make sure your pipeline role access policy grants the required permissions. Then, include this role in a data access policy and provide it permissions to create indexes, update indexes, describe indexes, and write documents within the collection. For instructions to complete each of these steps, see [Allowing pipelines to access collections](./pipeline-collection-access.html). + +#### Permissions to write to Amazon S3 or a dead-letter queue + +If you specify Amazon S3 as a sink destination for your pipeline, or if you enable a [dead-letter queue](https://opensearch.org/docs/latest/data-prepper/pipelines/dlq/) (DLQ), the pipeline role must allow it to access the S3 bucket that you specify as the destination. + +Attach a separate permissions policy to the pipeline role that provides DLQ access. At minimum, the role must be granted the `S3:PutObject` action on the bucket resource: @@ -159 +127,3 @@ You must use the same `sts_role_arn` in all pipeline components. Therefore, you -You can then specify the role within the pipeline's DLQ configuration: +## Ingestion role + +The ingestion role is an IAM role that allows external services to securely interact with and send data to an OpenSearch Ingestion pipeline. For push-based sources, such as Amazon Security Lake, this role must grant permissions to push data into the pipeline, including `osis:Ingest`. For pull-based sources, like Amazon S3, the role must enable OpenSearch Ingestion to assume it and access the data with the necessary permissions. @@ -160,0 +131 @@ You can then specify the role within the pipeline's DLQ configuration: +###### Topics @@ -162,9 +133 @@ You can then specify the role within the pipeline's DLQ configuration: - ... - sink: - opensearch: - dlq: