AWS opensearch-service documentation change
Summary
Restructured tutorial steps, automated pipeline role creation, updated IAM permissions, and simplified pipeline configuration process.
Security assessment
The changes introduce automated IAM role creation (with iam:CreateRole and iam:AttachPolicy permissions) and emphasize service-managed roles, reducing manual configuration errors. While this improves security practices, there's no evidence of addressing a specific vulnerability. The tutorial now explicitly mentions public access vs. VPC configuration, adding security-related guidance.
Diff
diff --git a/opensearch-service/latest/developerguide/osis-serverless-get-started.md b/opensearch-service/latest/developerguide/osis-serverless-get-started.md index 0ee259d51..4b8ca7e6c 100644 --- a//opensearch-service/latest/developerguide/osis-serverless-get-started.md +++ b//opensearch-service/latest/developerguide/osis-serverless-get-started.md @@ -5 +5 @@ -Required permissionsStep 1: Create the pipeline roleStep 2: Create a collectionStep 3: Create a pipelineStep 4: Ingest some sample dataRelated resources +Required permissionsStep 1: Create a collectionStep 2: Create a pipelineStep 3: Ingest some sample dataRelated resources @@ -13 +13 @@ For a tutorial that demonstrates how to ingest data into a provisioned OpenSearc -You'll complete the following steps in this tutorial: +You'll complete the following steps in this tutorial:. @@ -15 +15 @@ You'll complete the following steps in this tutorial: - 1. Create the pipeline role. + 1. Create a collection. @@ -17 +17 @@ You'll complete the following steps in this tutorial: - 2. Create a collection. + 2. Create a pipeline. @@ -19,3 +19 @@ You'll complete the following steps in this tutorial: - 3. Create a pipeline. - - 4. Ingest some sample data. + 3. Ingest some sample data. @@ -28,2 +25,0 @@ Within the tutorial, you'll create the following resources: - * A pipeline named `ingestion-pipeline-serverless` - @@ -32 +28 @@ Within the tutorial, you'll create the following resources: - * An IAM role named `PipelineRole` that the pipeline will assume in order to write to the collection + * A pipeline named `ingestion-pipeline-serverless` @@ -39 +35 @@ Within the tutorial, you'll create the following resources: -To complete this tutorial, you must have the correct IAM permissions. Your user or role must have an attached [identity-based policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security-iam-serverless-id-based-policies) with the following minimum permissions. These permissions allow you to create a pipeline role (`iam:Create*`), create or modify a collection (`aoss:*`), and work with pipelines (`osis:*`). +To complete this tutorial, your user or role must have an attached [identity-based policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security-iam-serverless-id-based-policies) with the following minimum permissions. These permissions allow you to create a pipeline role and attach a policy (`iam:Create*` and `iam:Attach*`), create or modify a collection (`aoss:*`), and work with pipelines (`osis:*`). @@ -41 +37 @@ To complete this tutorial, you must have the correct IAM permissions. Your user -In addition, the `iam:PassRole` permission is required on the pipeline role resource. This permission allows you to pass the pipeline role to OpenSearch Ingestion so that it can write data to the collection. +In addition, several IAM permissions are required in order to automatically create the pipeline role and pass it to OpenSearch Ingestion so that it can write data to the collection. @@ -52,0 +49 @@ In addition, the `iam:PassRole` permission is required on the pipeline role reso + "iam:Attach*", @@ -58 +55 @@ In addition, the `iam:PassRole` permission is required on the pipeline role reso - "arn:aws:iam::your-account-id:role/PipelineRole" + "arn:aws:iam::your-account-id:role/OpenSearchIngestion-PipelineRole" @@ -61,0 +59,2 @@ In addition, the `iam:PassRole` permission is required on the pipeline role reso + "iam:CreateRole", + "iam:AttachPolicy", @@ -68,69 +67 @@ In addition, the `iam:PassRole` permission is required on the pipeline role reso -## Step 1: Create the pipeline role - -First, create a role that the pipeline will assume in order to access the OpenSearch Serverless collection sink. You'll include this role within the pipeline configuration later in this tutorial. - -###### To create the pipeline role - - 1. Open the AWS Identity and Access Management console at [https://console.aws.amazon.com/iamv2/](https://console.aws.amazon.com/iamv2/ ). - - 2. Choose **Policies** , and then choose **Create policy**. - - 3. Select **JSON** and paste the following policy into the editor. Modify the collection ARN and name accordingly. - - { - "Version": "2012-10-17", - "Statement": [ - { - "Action": [ - "aoss:BatchGetCollection", - "aoss:APIAccessAll" - ], - "Effect": "Allow", - "Resource": "arn:aws:aoss:us-east-1:your-account-id:collection/collection-id" - }, - { - "Action": [ - "aoss:CreateSecurityPolicy", - "aoss:GetSecurityPolicy", - "aoss:UpdateSecurityPolicy" - ], - "Effect": "Allow", - "Resource": "*", - "Condition": { - "StringEquals": { - "aoss:collection": "collection-name" - } - } - } - ] - } - - 4. Choose **Next** , choose **Next** , and name your policy **collection-pipeline-policy**. - - 5. Choose **Create policy**. - - 6. Next, create a role and attach the policy to it. Choose **Roles** , and then choose **Create role**. - - 7. Choose **Custom trust policy** and paste the following policy into the editor: - - { - "Version":"2012-10-17", - "Statement":[ - { - "Effect":"Allow", - "Principal":{ - "Service":"osis-pipelines.amazonaws.com" - }, - "Action":"sts:AssumeRole" - } - ] - } - - 8. Choose **Next**. Then search for and select **collection-pipeline-policy** (which you just created). - - 9. Choose **Next** and name the role **PipelineRole**. - - 10. Choose **Create role**. - - - +## Step 1: Create a collection @@ -138,5 +69 @@ First, create a role that the pipeline will assume in order to access the OpenSe -Remember the Amazon Resource Name (ARN) of the role (for example, `arn:aws:iam::`your-account-id`:role/PipelineRole`). You'll need it when you create your pipeline. - -## Step 2: Create a collection - -Next, create a collection to ingest data into. We'll name the collection `ingestion-collection`. +First, create a collection to ingest data into. We'll name the collection `ingestion-collection`. @@ -182 +109 @@ Next, create a collection to ingest data into. We'll name the collection `ingest - "arn:aws:iam::your-account-id:role/PipelineRole", + "arn:aws:iam::your-account-id:role/OpenSearchIngestion-PipelineRole", @@ -189 +116 @@ Next, create a collection to ingest data into. We'll name the collection `ingest - 9. Replace the `Principal` elements. For the first principal, specify pipeline role that you created. For the second principal, specify a user or role that you can use to query the collection later. + 9. Modify the `Principal` elements to include your AWS account ID. For the second principal, specify a user or role that you can use to query the collection later. @@ -198,3 +125 @@ Next, create a collection to ingest data into. We'll name the collection `ingest -When the collection is active, note the OpenSearch endpoint under **Endpoint** (for example, `https://`{collection-id}`.us-east-1.aoss.amazonaws.com`). You'll need it when you create your pipeline. - -## Step 3: Create a pipeline +## Step 2: Create a pipeline @@ -202 +127 @@ When the collection is active, note the OpenSearch endpoint under **Endpoint** ( -Now that you have a collection and a role with the appropriate access rights, you can create a pipeline. +Now that you have a collection, you can create a pipeline. @@ -210 +135,7 @@ Now that you have a collection and a role with the appropriate access rights, yo - 3. Select the blank pipeline, then choose **Select blueprint**. + 3. Select the **Blank** pipeline, then choose **Select blueprint**. + + 4. In this tutorial, we'll create a simple pipeline that uses the [HTTP source](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http-source/) plugin. The plugin accepts log data in a JSON array format. We'll specify a single OpenSearch Serverless collection as the sink, and ingest all data into the `my_logs` index. + +In the **Source** menu, choose **HTTP**. For the **Path** , enter **/logs**. + + 5. For simplicity in this tutorial, we'll configure public access for the pipeline. For **Source network options** , choose **Public access**. For information about configuring VPC access, see [Configuring VPC access for Amazon OpenSearch Ingestion pipelines](./pipeline-security.html). @@ -212 +143 @@ Now that you have a collection and a role with the appropriate access rights, yo - 4. Name the pipeline **serverless-ingestion** and keep the capacity settings as their defaults. + 6. Choose **Next**. @@ -214 +145 @@ Now that you have a collection and a role with the appropriate access rights, yo - 5. In this tutorial, we'll create a simple sub-pipeline called `log-pipeline` that uses the [HTTP source](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/http-source/) plugin. The plugin accepts log data in a JSON array format. We'll specify a single OpenSearch Serverless collection as the sink, and ingest all data into the `my_logs` index. + 7. For **Processor** , enter **Date** and choose **Add**. @@ -216 +147 @@ Now that you have a collection and a role with the appropriate access rights, yo -Under **Pipeline configuration** , paste the following YAML configuration into the editor: + 8. Enable **From time received**. Leave all other settings as their defaults. @@ -218,17 +149 @@ Under **Pipeline configuration** , paste the following YAML configuration into t - version: "2" - log-pipeline: - source: - http: - path: "/${pipelineName}/test_ingestion_path" - processor: - - date: - from_time_received: true - destination: "@timestamp" - sink: - - opensearch: - hosts: [ "https://collection-id.us-east-1.aoss.amazonaws.com" ] - index: "my_logs" - aws: - sts_role_arn: "arn:aws:iam::your-account-id:role/PipelineRole" - region: "us-east-1" - **serverless: true** + 9. Choose **Next**. @@ -236 +151 @@ Under **Pipeline configuration** , paste the following YAML configuration into t - 6. Replace the `hosts` URL with the endpoint of the collection that you created in the previous section. Replace the `sts_role_arn` parameter with the ARN of `PipelineRole`. Optionally, modify the `region`. + 10. Configure sink details. For **OpenSearch resource type** , choose **Collection (Serverless)**. Then choose the OpenSearch Service collection that you created in the previous section. @@ -238 +153 @@ Under **Pipeline configuration** , paste the following YAML configuration into t - 7. Choose **Validate pipeline** and make sure that the validation succeeds. +Leave the network policy name as the default. For **Index name** , enter **my_logs**. OpenSearch Ingestion automatically creates this index in the collection if it doesn't already exist. @@ -240 +155 @@ Under **Pipeline configuration** , paste the following YAML configuration into t - 8. For simplicity in this tutorial, we'll configure public access for the pipeline. Under **Network** , choose **Public access**. + 11. Choose **Next**. @@ -242 +157 @@ Under **Pipeline configuration** , paste the following YAML configuration into t -For information about configuring VPC access, see [Configuring VPC access for Amazon OpenSearch Ingestion pipelines](./pipeline-security.html). + 12. Name the pipeline **ingestion-pipeline-serverless**. Leave the capacity settings as their defaults. @@ -244 +159 @@ For information about configuring VPC access, see [Configuring VPC access for Am - 9. Keep log publishing enabled in case you encounter any issues while completing this tutorial. For more information, see [Monitoring pipeline logs](./monitoring-pipeline-logs.html). + 13. For **Pipeline role** , select **Create and use a new service role**. The pipeline role provides the required permissions for a pipeline to write to the collection sink and read from pull-based sources. By selecting this option, you allow OpenSearch Ingestion to create the role for you, rather than manually creating it in IAM. For more information, see [Setting up roles and users in Amazon OpenSearch Ingestion](./pipeline-security-overview.html). @@ -246 +161 @@ For information about configuring VPC access, see [Configuring VPC access for Am -Specify the following log group name: `/aws/vendedlogs/OpenSearchIngestion/serverless-ingestion/audit-logs` + 14. For **Service role name suffix** , enter **PipelineRole**. In IAM, the role will have the format `arn:aws:iam::`your-account-id`:role/OpenSearchIngestion-**PipelineRole**`. @@ -248 +163 @@ Specify the following log group name: `/aws/vendedlogs/OpenSearchIngestion/serve - 10. Choose **Next**. Review your pipeline configuration and choose **Create pipeline**. The pipeline takes 5–10 minutes to become active. + 15. Choose **Next**. Review your pipeline configuration and choose **Create pipeline**. The pipeline takes 5–10 minutes to become active. @@ -253 +168 @@ Specify the following log group name: `/aws/vendedlogs/OpenSearchIngestion/serve -## Step 4: Ingest some sample data +## Step 3: Ingest some sample data @@ -265 +180 @@ First, get the ingestion URL from the **Pipeline settings** page: -Then, ingest some sample data. The following sample request uses [awscurl](https://github.com/okigan/awscurl) to send a single log file to the `my_logs` index: +Then, send some sample data to the ingestion path. The following sample request uses [awscurl](https://github.com/okigan/awscurl) to send a single log file to the pipeline: @@ -272 +187 @@ Then, ingest some sample data. The following sample request uses [awscurl](https - https://pipeline-endpoint.us-east-1.osis.amazonaws.com/log-pipeline/test_ingestion_path + https://pipeline-endpoint.us-east-1.osis.amazonaws.com/logs