AWS opensearch-service documentation change
Summary
Restructured documentation for Kinesis Data Streams integration with OpenSearch Ingestion, adding cross-account configuration details, pipeline blueprints, and enhanced security configuration examples
Security assessment
Added detailed IAM role configurations, KMS decryption policies, and cross-account resource policies but no evidence of addressing specific vulnerabilities. Enhances security documentation by explaining proper role setup and encryption handling.
Diff
diff --git a/opensearch-service/latest/developerguide/configure-client-kinesis.md b/opensearch-service/latest/developerguide/configure-client-kinesis.md index d5384369d..b0ded3866 100644 --- a//opensearch-service/latest/developerguide/configure-client-kinesis.md +++ b//opensearch-service/latest/developerguide/configure-client-kinesis.md @@ -5 +5 @@ -Connectivity to Amazon Kinesis Data StreamsStep 1: Configure the pipeline roleStep 2: Create the pipelineData consistency +Amazon Kinesis Data Streams as a sourceAmazon Kinesis Data Streams cross account as a source @@ -7 +7 @@ Connectivity to Amazon Kinesis Data StreamsStep 1: Configure the pipeline roleSt -# Using an OpenSearch Ingestion pipeline with Amazon Kinesis Data Streams +# Use an OpenSearch Ingestion pipeline with Amazon Kinesis Data Streams @@ -9 +9 @@ Connectivity to Amazon Kinesis Data StreamsStep 1: Configure the pipeline roleSt -You can use the [Kinesis](https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/kinesis/) plugin to stream data from Amazon Kinesis Data Streams to Amazon OpenSearch Service domains and OpenSearch Serverless collections. The pipeline pulls records from Amazon Kinesis and sends them to OpenSearch, and automatically generates indexes based on the stream name and current date. +Use an OpenSearch Ingestion pipeline with Amazon Kinesis Data Streams to ingest stream records data from multiple streams into Amazon OpenSearch Service domains and collections. The OpenSearch Ingestion pipeline incorporates the streaming ingestion infrastructure to provide a high-scale, low latency way to continuously ingest stream records from Kinesis. @@ -11 +11 @@ You can use the [Kinesis](https://opensearch.org/docs/latest/data-prepper/pipeli -## Connectivity to Amazon Kinesis Data Streams +###### Topics @@ -13 +13 @@ You can use the [Kinesis](https://opensearch.org/docs/latest/data-prepper/pipeli -You can use OpenSearch Ingestion pipelines to migrate data from Amazon Kinesis Data Streams with public configuration, which means that the domain DNS name can be publicly resolved. To do so, set up an OpenSearch Ingestion pipeline with Amazon Kinesis Data Streams as the source and OpenSearch Service or OpenSearch Serverless as the destination. This processes your streaming data from a self-managed source cluster to an AWS-managed destination domain or collection. + * Amazon Kinesis Data Streams as a source @@ -15,9 +15 @@ You can use OpenSearch Ingestion pipelines to migrate data from Amazon Kinesis D -### Prerequisites - -Before you create your OpenSearch Ingestion pipeline, perform the following steps: - - 1. Create an Amazon Kinesis data stream acting as a source. The stream should contain the data you want to ingest into OpenSearch Service. - - 2. Create an OpenSearch Service domain or OpenSearch Serverless collection where you want to migrate data to. For more information, see [Creating OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html#createdomains) and [Creating collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-manage.html#serverless-create). - - 3. Set up authentication on your Amazon Kinesis data stream with AWS Secrets Manager. Enable secrets rotation by following the steps in [Rotate AWS Secrets Manager secrets](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html). + * Amazon Kinesis Data Streams cross account as a source @@ -25 +16,0 @@ Before you create your OpenSearch Ingestion pipeline, perform the following step - 4. Attach a [resource-based policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-resource) to your domain or a [data access policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html) to your collection. These access policies allow OpenSearch Ingestion to write data from your self-managed cluster to your domain or collection. @@ -27 +17,0 @@ Before you create your OpenSearch Ingestion pipeline, perform the following step -The following sample domain access policy allows the pipeline role, which you create in the next step, to write data to a domain. Make sure that you update the `resource` with your own ARN. @@ -29 +18,0 @@ The following sample domain access policy allows the pipeline role, which you cr -JSON @@ -30,0 +20 @@ JSON +## Amazon Kinesis Data Streams as a source @@ -32 +22 @@ JSON -**** +With the following procedure, you'll learn how to set up an OpenSearch Ingestion pipeline that uses Amazon Kinesis Data Streams as the data source. This section covers the necessary prerequisites, such as creating an OpenSearch Service domain or an OpenSearch Serverless Collection, and walking through the steps to configure the pipeline role and create the pipeline. @@ -33,0 +24 @@ JSON +### Prerequisites @@ -35,28 +26 @@ JSON - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam::account-id:role/pipeline-role" - }, - "Action": [ - "es:DescribeDomain", - "es:ESHttp*" - ], - "Resource": [ - "arn:aws:es:region:account-id:domain/domain-name" - ] - } - ] - } - - -To create an IAM role with the correct permissions to access write data to the collection or domain, see [Setting up roles and users in Amazon OpenSearch Ingestion](./pipeline-security-overview.html). - - - - -## Step 1: Configure the pipeline role - -After you have your Amazon Kinesis Data Streams pipeline prerequisites set up, [configure the pipeline role](./pipeline-security-overview.html#pipeline-security-sink) that you want to use in your pipeline configuration, and add permission to write to an OpenSearch Service domain or OpenSearch Serverless collection, as well as permission to read secrets from Secrets Manager. +To set up your pipeline, you need one or more active Kinesis Data Streams. These streams must be either receiving records or ready to receive records from other sources. For more information, see [Overview of OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-getting-started-tutorials.html). @@ -64 +28 @@ After you have your Amazon Kinesis Data Streams pipeline prerequisites set up, [ -The following permission is needed to write to an Amazon S3 bucket, domain, and collection: +###### To set up your pipeline @@ -66 +30 @@ The following permission is needed to write to an Amazon S3 bucket, domain, and -JSON + 1. ###### Create an OpenSearch Service domain or an OpenSearch Serverless collection @@ -67,0 +32 @@ JSON +To create a domain or a collection, see [Getting started with OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-getting-started-tutorials.html). @@ -69 +34 @@ JSON -**** +To create an IAM role with the correct permissions to access write data to the collection or domain, see [Resource-based policies](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-resource). @@ -70,0 +36 @@ JSON + 2. ###### Configure the pipeline role with permissions @@ -71,0 +38 @@ JSON +[Set up the pipeline role](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-security-overview.html#pipeline-security-sink) that you want to use in your pipeline configuration and add the following permissions to it. Replace the `placeholder values` with your own information. @@ -92 +59 @@ JSON - "arn:aws:kinesis:region:account-id:stream/stream-name" + "arn:aws:kinesis:AWS Region:account-id:stream/stream-name" @@ -99,8 +66 @@ JSON -If server-side encryption is enabled for streams, the following KMS policy will decrypt the stream records: - -JSON - - -**** - - +If server-side encryption is enabled on the streams, the following AWS KMS policy allows to decrypt the records. Replace the `placeholder values` with your own information. @@ -118 +78 @@ JSON - "Resource": "arn:aws:kms:region:account-id:key/key-id" + "Resource": "arn:aws:kinesis:AWS Region:account-id:key/key-id" @@ -124 +84 @@ JSON -In order for a pipeline to write data to a domain, the domain must have a [domain-level access policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-resource) that allows the **sts_role_arn** pipeline role to access it. The following sample domain access policy allows the pipeline role named `pipeline-role` , which you created in the previous step, to write data to the domain named `ingestion-domain` : +In order for a pipeline to write data to a domain, the domain must have a [domain-level access policy](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-types-resource) that allows the **sts_role_arn** pipeline role to access it. @@ -125,0 +86 @@ In order for a pipeline to write data to a domain, the domain must have a [domai +The following example is a domain access policy that allows the pipeline role created in the previous step (`pipeline-role`), to write data to the `ingestion-domain` domain. Replace the `placeholder values` with your own information. @@ -132 +93 @@ In order for a pipeline to write data to a domain, the domain must have a [domai - "AWS": "arn:aws:iam::account-id:role/pipeline-role" + "AWS": "arn:aws:iam::your-account-id:role/pipeline-role" @@ -135 +96 @@ In order for a pipeline to write data to a domain, the domain must have a [domai - "Resource": "arn:aws:es:region:account-id:domain/domain-name/*" + "Resource": "arn:aws:es:AWS Region:account-id:domain/domain-name/*" @@ -140 +100,0 @@ In order for a pipeline to write data to a domain, the domain must have a [domai -## Step 2: Create the pipeline @@ -142 +102,98 @@ In order for a pipeline to write data to a domain, the domain must have a [domai -You can then configure an OpenSearch Ingestion pipeline which specifying Amazon Kinesis as the source. The metadata attributes available are: + 3. ###### Create the pipeline + +Configure an OpenSearch Ingestion pipeline specifying **Kinesis-data-streams** as the source. You can locate a ready made blueprint available on the OpenSearch Ingestion Console for creating such a pipeline. (Optional) To create the pipeline using the AWS CLI, you can use a blueprint named "**`AWS-KinesisDataStreamsPipeline`** ". Replace the `placeholder values` with your own information. + + version: "2" + kinesis-pipeline: + source: + kinesis_data_streams: + acknowledgments: true + codec: + # Based on whether kinesis records are aggregated or not, you could choose json, newline or ndjson codec for processing the records. + # JSON codec supports parsing nested CloudWatch Events into individual log entries that will be written as documents into OpenSearch. + # json: + # key_name: "logEvents" + # These keys contain the metadata sent by CloudWatch Subscription Filters + # in addition to the individual log events: + # include_keys: [ 'owner', 'logGroup', 'logStream' ] + newline: + streams: + - stream_name: "stream name" + # Enable this if ingestion should start from the start of the stream. + # initial_position: "EARLIEST" + # checkpoint_interval: "PT5M" + # Compression will always be gzip for CloudWatch, but will vary for other sources: + # compression: "gzip" + - stream_name: "stream name" + # Enable this if ingestion should start from the start of the stream. + # initial_position: "EARLIEST" + # checkpoint_interval: "PT5M" + # Compression will always be gzip for CloudWatch, but will vary for other sources: + # compression: "gzip" + + # buffer_timeout: "1s" + # records_to_accumulate: 100 + # Change the consumer strategy to "polling". Default consumer strategy will use enhanced "fan-out" supported by KDS. + # consumer_strategy: "polling" + # if consumer strategy is set to "polling", enable the polling config below. + # polling: + # max_polling_records: 100 + # idle_time_between_reads: "250ms" + aws: + # Provide the Role ARN with access to Amazon Kinesis Data Streams. This role should have a trust relationship with osis-pipelines.amazonaws.com + sts_role_arn: "arn:aws:iam::111122223333:role/Example-Role" + # Provide the AWS Region of the Data Stream. + region: "us-east-1" + + sink: + - opensearch: + # Provide an Amazon OpenSearch Serverless domain endpoint + hosts: [ "https://search-mydomain-1a2a3a4a5a6a7a8a9a0a9a8a7a.us-east-1.es.amazonaws.com" ] + index: "index_${getMetadata(\"stream_name\")}" + # Ensure adding unique document id as a combination of the metadata attributes available. + document_id: "${getMetadata(\"partition_key\")}_${getMetadata(\"sequence_number\")}_${getMetadata(\"sub_sequence_number\")}" + aws: + # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com + sts_role_arn: "arn:aws:iam::111122223333:role/Example-Role" + # Provide the AWS Region of the domain. + region: "us-east-1" + # Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection + serverless: false + # serverless_options: + # Specify a name here to create or update network policy for the serverless collection + # network_policy_name: "network-policy-name" + # Enable the 'distribution_version' setting if the OpenSearch Serverless domain is of version Elasticsearch 6.x + # distribution_version: "es6" + # Enable and switch the 'enable_request_compression' flag if the default compression setting is changed in the domain. See https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gzip.html + # enable_request_compression: true/false + # Optional: Enable the S3 DLQ to capture any failed requests in an S3 bucket. Delete this entire block if you don't want a DLQ. + dlq: + s3: + # Provide an S3 bucket + bucket: "your-dlq-bucket-name" + # Provide a key path prefix for the failed requests + # key_path_prefix: "kinesis-pipeline/logs/dlq" + # Provide the region of the bucket. + region: "us-east-1" + # Provide a Role ARN with access to the bucket. This role should have a trust relationship with osis-pipelines.amazonaws.com + sts_role_arn: "arn:aws:iam::111122223333:role/Example-Role" + + +###### Configuration options