AWS opensearch-service documentation change
Summary
Updated documentation for DocumentDB plugin integration, including simplified explanations of snapshot processing, corrected IAM policy syntax examples, and updated internal links
Security assessment
Changes focus on documentation clarity and syntax corrections (removing placeholder braces in ARNs). No evidence of addressing security vulnerabilities or adding new security features. IAM policy updates are syntax fixes rather than security enhancements.
Diff
diff --git a/opensearch-service/latest/developerguide/configure-client-docdb.md b/opensearch-service/latest/developerguide/configure-client-docdb.md index ffd6c6a43..472192824 100644 --- a/opensearch-service/latest/developerguide/configure-client-docdb.md +++ b/opensearch-service/latest/developerguide/configure-client-docdb.md @@ -9 +9 @@ PrerequisitesStep 1: Configure the pipeline roleStep 2: Create the pipelineData -You can use an OpenSearch Ingestion pipeline with Amazon DocumentDB to stream document changes (such as create, update, and delete) to Amazon OpenSearch Service domains and collections. The OpenSearch Ingestion pipeline can leverage change data capture (CDC) mechanisms, if available on your Amazon DocumentDB cluster, or API polling to provide a high-scale, low-latency way to continuously stream data from a Amazon DocumentDB cluster. +You can use the [DocumentDB](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/documentdb/) plugin to stream document changes, such as creates, updates, and deletes, to Amazon OpenSearch Service. The pipeline supports change data capture (CDC), if available, or API polling for high-scale, low-latency streaming. @@ -11 +11 @@ You can use an OpenSearch Ingestion pipeline with Amazon DocumentDB to stream do -There are two ways that you can use Amazon DocumentDB as a source to process data—with and without a _full initial snapshot_. +You can process data with or without a full initial snapshot. A full snapshot captures an entire Amazon DocumentDB collection and uploads it to Amazon S3. The pipeline then sends the data to one or more OpenSearch indexes. After it ingests the snapshot, the pipeline synchronizes ongoing changes to maintain consistency and eventually catches up to near real-time updates. @@ -13 +13 @@ There are two ways that you can use Amazon DocumentDB as a source to process dat -A full initial snapshot is a bulk query of an entire Amazon DocumentDB collection. Amazon DocumentDB uploads this snapshot to Amazon S3. From there, an OpenSearch Ingestion pipeline sends it to one index in a domain, or partitions it to multiple indexes in a domain. To keep the data in Amazon DocumentDB and OpenSearch consistent, the pipeline syncs all of the create, update, and delete events in the Amazon DocumentDB collection with the documents saved in the OpenSearch index or indexes. +If you already have a full snapshot from another source, or only need to process new events, you can stream without a snapshot. In this case, the pipeline reads directly from Amazon DocumentDB change streams without an initial bulk load. @@ -15,5 +15 @@ A full initial snapshot is a bulk query of an entire Amazon DocumentDB collectio -When you use a full initial snapshot, your OpenSearch Ingestion pipeline first ingests the snapshot and then starts reading data from Amazon DocumentDB change streams. It eventually catches up and maintains near real-time data consistency between Amazon DocumentDB and OpenSearch. - -You can also use the OpenSearch Ingestion integration with Amazon DocumentDB to stream events without a snapshot. Choose this option if you already have a full snapshot from some other mechanism, or if you just want to stream current events from a Amazon DocumentDB collection with change streams. - -With both of these options, you must [enable a change stream](https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html#change_streams-enabling) on your Amazon DocumentDB collection if you enable a stream in your in pipeline configuration. If you only use full load or export, you don't need to enable a change stream. +If you enable streaming, you must [enable a change stream](https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html#change_streams-enabling) on your Amazon DocumentDB collection. However, if you only perform a full load or export, you don’t need a change stream. @@ -25 +21 @@ Before you create your OpenSearch Ingestion pipeline, perform the following step - 1. Create a Amazon DocumentDB cluster with permission to read data by following the steps in [Create an Amazon DocumentDB cluster](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html#cloud9-cluster) in the _Amazon DocumentDB Developer Guide_. If you use CDC infrastructure, ensure that you configure your Amazon DocumentDB cluster to publish change streams. + 1. Create a Amazon DocumentDB cluster with permission to read data by following the steps in [Create an Amazon DocumentDB cluster](https://docs.aws.amazon.com/documentdb/latest/developerguide/get-started-guide.html#cloud9-cluster) in the _Amazon DocumentDB Developer Guide_. If you use CDC infrastructure, configure your Amazon DocumentDB cluster to publish change streams. @@ -35 +31 @@ Before you create your OpenSearch Ingestion pipeline, perform the following step - 6. Create an OpenSearch Service domain or OpenSearch Serverless collection. For more information, see [Creating OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html#createdomains) and [Creating collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-manage.html#serverless-create). + 6. Create an OpenSearch Service domain or OpenSearch Serverless collection. For more information, see [ Creating OpenSearch Service domains](./createupdatedomains.html#createdomains) and [Creating collections](./serverless-manage.html#serverless-create). @@ -47 +43 @@ The following sample domain access policy allows the pipeline role, which you cr - "AWS": "arn:aws:iam::{pipeline-account-id}:role/pipeline-role" + "AWS": "arn:aws:iam::pipeline-account-id:role/pipeline-role" @@ -54 +50 @@ The following sample domain access policy allows the pipeline role, which you cr - "arn:aws:es:{region}:{account-id}:domain/domain-name" + "arn:aws:es:region:account-id:domain/domain-name" @@ -60 +56 @@ The following sample domain access policy allows the pipeline role, which you cr -To create an IAM role with the correct permissions to access write data to the collection or domain, see [Required permissions for domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-get-started.html#osis-get-started-permissions) and [Required permissions for collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-serverless-get-started.html#osis-serverless-get-started-permissions). +To create an IAM role with the correct permissions to access write data to the collection or domain, see [Setting up roles and users in Amazon OpenSearch Ingestion](./pipeline-security-overview.html). @@ -80 +76 @@ After you have your Amazon DocumentDB pipeline prerequisites set up, [configure - "arn:aws:s3:::{s3_bucket}" + "arn:aws:s3:::s3-bucket" @@ -84 +80 @@ After you have your Amazon DocumentDB pipeline prerequisites set up, [configure - "s3:prefix": "{s3_prefix}/*" + "s3:prefix": "s3-prefix/*" @@ -97 +93 @@ After you have your Amazon DocumentDB pipeline prerequisites set up, [configure - "arn:aws:s3:::{s3_bucket}/{s3_prefix}/*" + "arn:aws:s3:::s3-bucket/s3-prefix/*" @@ -106 +102 @@ After you have your Amazon DocumentDB pipeline prerequisites set up, [configure - "Resource": ["arn:aws:secretsmanager:{region}:{account-id}:secret:secret-name"] + "Resource": ["arn:aws:secretsmanager:region:account-id:secret:secret-name"] @@ -120,3 +116,3 @@ After you have your Amazon DocumentDB pipeline prerequisites set up, [configure - "arn:aws:ec2:*:{account-id}:network-interface/*", - "arn:aws:ec2:*:{account-id}:subnet/*", - "arn:aws:ec2:*:{account-id}:security-group/*" + "arn:aws:ec2:*:account-id:network-interface/*", + "arn:aws:ec2:*:account-id:subnet/*", + "arn:aws:ec2:*:account-id:security-group/*" @@ -171 +167 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - sts_role_arn: "arn:aws:iam::{account-id}:role/pipeline-role" + sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role" @@ -194 +190 @@ You can then configure an OpenSearch Ingestion pipeline like the following, whic - sts_role_arn: "arn:aws:iam::{account-id}:role/pipeline-role" + sts_role_arn: "arn:aws:iam::account-id:role/pipeline-role"