AWS opensearch-service documentation change
Summary
Restructured documentation to emphasize pipeline role creation and network/data access configuration. Added details about automatic role creation, simplified network policy examples, and removed redundant pipeline creation steps.
Security assessment
The changes clarify security best practices (least privilege IAM roles, network policy configuration) and document how OpenSearch Ingestion automatically manages VPC endpoints and network policies. However, there is no evidence of addressing a specific security vulnerability or incident.
Diff
diff --git a/opensearch-service/latest/developerguide/pipeline-collection-access.md b/opensearch-service/latest/developerguide/pipeline-collection-access.md index 2686bc45f..0c212982d 100644 --- a//opensearch-service/latest/developerguide/pipeline-collection-access.md +++ b//opensearch-service/latest/developerguide/pipeline-collection-access.md @@ -5 +5 @@ -Providing network access to pipelinesStep 1: Create a pipeline roleStep 2: Create a collectionStep 3: Create a pipeline +Step 1: Create the pipeline roleStep 2: Configure data and network access for the collection @@ -9,26 +9 @@ Providing network access to pipelinesStep 1: Create a pipeline roleStep 2: Creat -An Amazon OpenSearch Ingestion pipeline can write to an OpenSearch Serverless public collection or VPC collection. To provide access to the collection, you configure an AWS Identity and Access Management (IAM) pipeline role with a permissions policy that grants access to the collection. Before you specify the role in your pipeline configuration, you must configure it with an appropriate trust relationship, and then grant it data access permissions through a data access policy. - -During pipeline creation, OpenSearch Ingestion creates an AWS PrivateLink connection between the pipeline and the OpenSearch Serverless collection. All traffic from the pipeline goes through this VPC endpoint and is routed to the collection. In order to reach the collection, the endpoint must be granted access to the collection through a network access policy. - - - -###### Topics - - * Providing network access to pipelines - - * Step 1: Create a pipeline role - - * Step 2: Create a collection - - * Step 3: Create a pipeline - - - - -## Providing network access to pipelines - -Each collection that you create in OpenSearch Serverless has at least one network access policy associated with it. Network access policies determine whether the collection is accessible over the internet from public networks, or whether it must be accessed privately. For more information about network policies, see [Network access for Amazon OpenSearch Serverless](./serverless-network.html). - -Within a network access policy, you can only specify OpenSearch Serverless-managed VPC endpoints. For more information, see [Access Amazon OpenSearch Serverless using an interface endpoint (AWS PrivateLink)](./serverless-vpc.html). However, in order for the pipeline to write to the collection, the policy must also grant access to the VPC endpoint that OpenSearch Ingestion automatically creates between the pipeline and the collection. Therefore, when you create a pipeline that has an OpenSearch Serverless collection sink, you must provide the name of the associated network policy using the `network_policy_name` option. - -For example: +An Amazon OpenSearch Ingestion pipeline can write to an OpenSearch Serverless public collection or VPC collection. To provide access to the collection, you configure an AWS Identity and Access Management (IAM) pipeline role with a permissions policy that grants access to the collection. The pipeline assumes this role in order to sign requests to the OpenSearch Serverless collection sink. @@ -35,0 +11 @@ For example: +###### Important @@ -37,9 +13 @@ For example: - ... - sink: - - opensearch: - hosts: [ "https://collection-id.region.aoss.amazonaws.com" ] - index: "my-index" - aws: - serverless: true - serverless_options: - **network_policy_name** : "network-policy-name" +You can choose to manually create the pipeline role, or you can have OpenSearch Ingestion create it for you during pipeline creation. If you choose automatic role creation, OpenSearch Ingestion adds all required permissions to the pipeline role access policy based on the source and sink that you choose. It creates a pipeline role in IAM with the prefix `OpenSearchIngestion-` and the suffix that you enter. For more information, see [Pipeline role](./pipeline-security-overview.html#pipeline-security-sink). @@ -47 +15 @@ For example: -During pipeline creation, OpenSearch Ingestion checks for the existence of the specified network policy. If it doesn't exist, OpenSearch Ingestion creates it. If it does exist, OpenSearch Ingestion updates it by adding a new rule to it. The rule grants access to the VPC endpoint that connects the pipeline and the collection. +If you have OpenSearch Ingestion create the pipeline role for you, you still need to include the role in the collection's data access policy, either before or after you create the pipeline. See step 2 for instructions. @@ -49 +17 @@ During pipeline creation, OpenSearch Ingestion checks for the existence of the s -For example: +During pipeline creation, OpenSearch Ingestion creates an AWS PrivateLink connection between the pipeline and the OpenSearch Serverless collection. All traffic from the pipeline goes through this VPC endpoint and is routed to the collection. In order to reach the collection, the endpoint must be granted access to the collection through a network access policy. @@ -50,0 +19 @@ For example: + @@ -52,14 +21 @@ For example: - { - "Rules":[ - { - "Resource":[ - "collection/my-collection" - ], - "ResourceType":"collection" - } - ], - "SourceVPCEs":[ - "vpce-0c510712627e27269" # The ID of the VPC endpoint that OpenSearch Ingestion creates between the pipeline and collection - ], - "Description":"Created by Data Prepper" - } +###### Topics @@ -67 +23 @@ For example: -In the console, any rules that OpenSearch Ingestion adds to your network policies are named **Created by Data Prepper** : + * Step 1: Create the pipeline role @@ -69 +25 @@ In the console, any rules that OpenSearch Ingestion adds to your network policie - + * Step 2: Configure data and network access for the collection @@ -71 +26,0 @@ In the console, any rules that OpenSearch Ingestion adds to your network policie -###### Note @@ -73 +27,0 @@ In the console, any rules that OpenSearch Ingestion adds to your network policie -In general, a rule that specifies public access for a collection overrides a rule that specifies private access. Therefore, if the policy already had _public_ access configured, this new rule that OpenSearch Ingestion adds doesn't actually change the behavior of the policy. For more information, see [Policy precedence](./serverless-network.html#serverless-network-precedence). @@ -75 +28,0 @@ In general, a rule that specifies public access for a collection overrides a rul -If you stop or delete the pipeline, OpenSearch Ingestion deletes the VPC endpoint between the pipeline and the collection. It also modifies the network policy to remove the VPC endpoint from the list of allowed endpoints. If you restart the pipeline, it recreates the VPC endpoint and re-updates the network policy with the endpoint ID. @@ -77 +30 @@ If you stop or delete the pipeline, OpenSearch Ingestion deletes the VPC endpoin -## Step 1: Create a pipeline role +## Step 1: Create the pipeline role @@ -79 +32 @@ If you stop or delete the pipeline, OpenSearch Ingestion deletes the VPC endpoin -The role that you specify in the **sts_role_arn** parameter of a pipeline configuration must have an attached permissions policy that allows it to send data to the collection sink. It must also have a trust relationship that allows OpenSearch Ingestion to assume the role. For instructions on how to attach a policy to a role, see [Adding IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#add-policies-console) in the _IAM User Guide_. +The pipeline role must have an attached permissions policy that allows it to send data to the collection sink. It must also have a trust relationship that allows OpenSearch Ingestion to assume the role. For instructions on how to attach a policy to a role, see [Adding IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#add-policies-console) in the _IAM User Guide_. @@ -81 +34 @@ The role that you specify in the **sts_role_arn** parameter of a pipeline config -The following sample policy demonstrates the [least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) that you can provide in a pipeline configuration's **sts_role_arn** role for it to write to collections: +The following sample policy demonstrates the [least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) that you can provide in a pipeline role access policy for it to write to collections: @@ -118 +71 @@ The role must have the following [trust relationship](https://docs.aws.amazon.co -## Step 2: Create a collection +## Step 2: Configure data and network access for the collection @@ -152 +105 @@ Create a [data access policy](https://docs.aws.amazon.com/opensearch-service/lat -In the `Principal` element, specify the Amazon Resource Name (ARN) of the pipeline role that you created in the previous step. +In the `Principal` element, specify the Amazon Resource Name (ARN) of the pipeline role. @@ -156 +109,7 @@ In the `Principal` element, specify the Amazon Resource Name (ARN) of the pipeli -Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-network.html) for the collection. You can ingest data into a public collection or a VPC collection. For example, the following policy provides access to a single OpenSearch Serverless-managed VPC endpoint: +Each collection that you create in OpenSearch Serverless has at least one network access policy associated with it. Network access policies determine whether the collection is accessible over the internet from public networks, or whether it must be accessed privately. For more information about network policies, see [Network access for Amazon OpenSearch Serverless](./serverless-network.html). + +Within a network access policy, you can only specify OpenSearch Serverless-managed VPC endpoints. For more information, see [Access Amazon OpenSearch Serverless using an interface endpoint (AWS PrivateLink)](./serverless-vpc.html). However, in order for the pipeline to write to the collection, the policy must also grant access to the VPC endpoint that OpenSearch Ingestion automatically creates between the pipeline and the collection. Therefore, if you choose an OpenSearch Serverless collection as the destination sink for a pipeline, you must enter the name of the associated network policy in the **Network policy name** field. + +During pipeline creation, OpenSearch Ingestion checks for the existence of the specified network policy. If it doesn't exist, OpenSearch Ingestion creates it. If it does exist, OpenSearch Ingestion updates it by adding a new rule to it. The rule grants access to the VPC endpoint that connects the pipeline and the collection. + +For example: @@ -159 +117,0 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - [ @@ -161 +118,0 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - "Description":"Rule 1", @@ -164 +120,0 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - "ResourceType":"collection", @@ -166,2 +122,3 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - "collection/collection-name" - ] + "collection/my-collection" + ], + "ResourceType":"collection" @@ -170 +126,0 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - "AllowFromPublic": false, @@ -172,2 +128,3 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - "vpce-050f79086ee71ac05" - ] + "vpce-0c510712627e27269" # The ID of the VPC endpoint that OpenSearch Ingestion creates between the pipeline and collection + ], + "Description":"Created by Data Prepper" @@ -175,20 +131,0 @@ Create a [network access policy](https://docs.aws.amazon.com/opensearch-service/ - ] - -###### Important - -You must specify the name of the network policy within the `network_policy_name` option in the pipeline configuration. At the time of pipeline creation, OpenSearch Ingestion updates this network policy to allow access to the VPC endpoint that it automatically creates between the pipeline and the collection. See step 3 for an example pipeline configuration. For more information, see Providing network access to pipelines. - -## Step 3: Create a pipeline - -Finally, create a pipeline in which you specify the pipeline role and collection details. The pipeline assumes this role in order to sign requests to the OpenSearch Serverless collection sink. - -Make sure to do the following: - - * For the `hosts` option, specify the endpoint of the collection that you created in step 2. - - * For the `sts_role_arn` option, specify the Amazon Resource Name (ARN) of the pipeline role that you created in step 1. - - * Set the `serverless` option to `true`. - - * Set the `network_policy_name` option to the name of the network policy attached to the collection. OpenSearch Ingestion automatically updates this network policy to allow access from the VPC that it creates between the pipeline and the collection. For more information, see Providing network access to pipelines. - @@ -195,0 +133 @@ Make sure to do the following: +In the console, any rules that OpenSearch Ingestion adds to your network policies are named **Created by Data Prepper** : @@ -196,0 +135 @@ Make sure to do the following: + @@ -197,0 +137 @@ Make sure to do the following: +###### Note @@ -199,19 +139 @@ Make sure to do the following: - version: "2" - log-pipeline: - source: - http: - path: "/log/ingest" - processor: - - date: - from_time_received: true - destination: "@timestamp" - sink: - - opensearch: - hosts: [ "https://collection-id.region.aoss.amazonaws.com" ] - index: "my-index" - aws: - serverless: true - serverless_options: - network_policy_name: "network-policy-name" # If the policy doesn't exist, a new policy is created. - region: "us-east-1" - sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role" +In general, a rule that specifies public access for a collection overrides a rule that specifies private access. Therefore, if the policy already had _public_ access configured, this new rule that OpenSearch Ingestion adds doesn't actually change the behavior of the policy. For more information, see [Policy precedence](./serverless-network.html#serverless-network-precedence). @@ -219 +141 @@ Make sure to do the following: -For a full reference of required and unsupported parameters, see [Supported plugins and options for Amazon OpenSearch Ingestion pipelines](./pipeline-config-reference.html). +If you stop or delete the pipeline, OpenSearch Ingestion deletes the VPC endpoint between the pipeline and the collection. It also modifies the network policy to remove the VPC endpoint from the list of allowed endpoints. If you restart the pipeline, it recreates the VPC endpoint and re-updates the network policy with the endpoint ID.