AWS AmazonCloudWatch documentation change
Summary
Restructured CloudWatch pipelines documentation to emphasize new features like logs management integration, automatic categorization, and third-party support. Updated pipeline components, pricing details, region availability, and operational limits. Added clarification about AWS Secrets Manager integration for credentials.
Security assessment
The change adds documentation for AWS Secrets Manager integration under 'Extensions', which is a security feature for credential management. However, there is no evidence of a specific security vulnerability being addressed. The note about avoiding PII in pipeline definitions was moved but not introduced in this change.
Diff
diff --git a/AmazonCloudWatch/latest/monitoring/cloudwatch-pipelines.md b/AmazonCloudWatch/latest/monitoring/cloudwatch-pipelines.md index 2fed624ba..e7a405d8f 100644 --- a//AmazonCloudWatch/latest/monitoring/cloudwatch-pipelines.md +++ b//AmazonCloudWatch/latest/monitoring/cloudwatch-pipelines.md @@ -4,0 +5,2 @@ +Pipeline componentsPricingRegion availability + @@ -7 +9 @@ -A telemetry pipeline collects, processes, and routes data such as logs, metrics, and traces from various sources to different destinations. CloudWatch pipelines provides a centralized way to collect data from AWS services, third-party applications, and custom sources. The pipeline processes and transforms data using a rich set of processors, converts data into standard formats like Open Cybersecurity Schema Framework (OCSF), and routes processed data to CloudWatch Logs. +CloudWatch pipelines is a fully managed data collector that ingests, transforms, and routes log data from AWS services, third-party applications, and custom sources to CloudWatch. Built-in processors let you enrich, filter, and standardize logs into formats like OCSF—without managing infrastructure or third-party tools. @@ -9 +11 @@ A telemetry pipeline collects, processes, and routes data such as logs, metrics, -You can collect data from a wide range of sources, including AWS services, third-party applications, and your own custom sources. This data is then processed and transformed using processors that can standardize formats, filter unnecessary information, and enrich the data with additional context. This allows you to convert varied data formats into standardized schemas such as OCSF, enabling unified analysis across all your data sources. +CloudWatch pipelines is fully integrated with the [logs management experience](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/data-source-discovery-management.html), enabling you to consistently process and enrich log data across related log groups via data-source and data-type specification. This unlocks use cases such as: @@ -11 +13 @@ You can collect data from a wide range of sources, including AWS services, third -Throughout the entire pipeline, your data remains protected with transport layer encryption, ensuring security and compliance requirements are met. + * **Automatic log categorization** – Logs processed through pipelines are automatically tagged with data source information, enabling service-centric discovery and querying across your infrastructure @@ -13 +15 @@ Throughout the entire pipeline, your data remains protected with transport layer -###### Note + * **Expanding third-party support** – Aggregate and normalize logs from a growing library of third-party sources for unified analytics and compliance @@ -15 +16,0 @@ Throughout the entire pipeline, your data remains protected with transport layer -When configuring pipelines, remember that pipeline definitions are not encrypted and should never include sensitive data, such as personally identifiable information (PII). @@ -17 +17,0 @@ When configuring pipelines, remember that pipeline definitions are not encrypted -Each pipeline consists of three main components: @@ -19 +18,0 @@ Each pipeline consists of three main components: - 1. **Source** – Defines where your data comes from (AWS services, third-party applications, or custom sources) @@ -21 +20 @@ Each pipeline consists of three main components: - 2. **Processors** – Optionally, configure how your data is transformed, filtered, or enriched +Output from pipelines is fully compatible with CloudWatch Logs features including Logs Insights queries, Logs Anomaly Detection, and Live Tail. CloudWatch pipelines works with both Standard and Infrequent Access log classes and is backwards compatible with Log Transformers. @@ -23 +22 @@ Each pipeline consists of three main components: -###### Note +To get started with CloudWatch pipelines, visit [pipelines within the CloudWatch ingestion page](https://console.aws.amazon.com/cloudwatch/home?#/telemetry-config:pipelines?useCase=All) in the CloudWatch console. @@ -25,3 +24 @@ Each pipeline consists of three main components: -Adding processors leads to mutation of the log events and original (raw) logs are not retained. - - 3. **Sink** – Specify where your processed data should be delivered +###### Note @@ -28,0 +26 @@ Adding processors leads to mutation of the log events and original (raw) logs ar +Be aware of the following limits that apply to CloudWatch pipelines @@ -29,0 +28 @@ Adding processors leads to mutation of the log events and original (raw) logs ar + * Maximum number of pipelines per account: 330 @@ -30,0 +30 @@ Adding processors leads to mutation of the log events and original (raw) logs ar + * Up to 300 pipelines for collecting data from CloudWatch Logs @@ -32 +32 @@ Adding processors leads to mutation of the log events and original (raw) logs ar -To get started with CloudWatch pipelines: + * Up to 30 pipelines for collecting data from other sources @@ -34 +33,0 @@ To get started with CloudWatch pipelines: - 1. Sign in to the AWS Management Console @@ -36 +34,0 @@ To get started with CloudWatch pipelines: - 2. Navigate to CloudWatch. @@ -38 +35,0 @@ To get started with CloudWatch pipelines: - 3. Choose **Ingestion** from the navigation panel and then select the **Pipelines** tab. @@ -40 +37 @@ To get started with CloudWatch pipelines: - 4. Choose **Create pipeline**. +## Pipeline components @@ -41,0 +39 @@ To get started with CloudWatch pipelines: +Each pipeline consists of the following components: @@ -42,0 +41 @@ To get started with CloudWatch pipelines: + * **Source** – Defines where data originates from (Amazon S3 buckets, CloudWatch Logs, third-party integration). Each pipeline must have exactly one source. @@ -43,0 +43 @@ To get started with CloudWatch pipelines: + * **Processors** (optional) – Transform, parse, and enrich log data as it flows through the pipeline. Processors are applied sequentially in the order they are defined. @@ -45 +45 @@ To get started with CloudWatch pipelines: -###### Note + * **Sink** – Defines the destination where processed log data is sent. Each pipeline must have exactly one sink. @@ -47 +47 @@ To get started with CloudWatch pipelines: -Be aware of the following limits that apply to CloudWatch pipelines + * **Extensions** (optional) – Provide additional functionality such as AWS Secrets Manager integration for credential management. @@ -49 +48,0 @@ Be aware of the following limits that apply to CloudWatch pipelines - * Maximum number of pipelines per account: 330 @@ -51 +49,0 @@ Be aware of the following limits that apply to CloudWatch pipelines - * Up to 300 pipelines for collecting data from CloudWatch Logs @@ -53 +50,0 @@ Be aware of the following limits that apply to CloudWatch pipelines - * Up to 30 pipelines for collecting data from other sources @@ -54,0 +52 @@ Be aware of the following limits that apply to CloudWatch pipelines +Throughout the entire pipeline, your data remains protected with transport layer encryption, ensuring security and compliance requirements are met. @@ -55,0 +54 @@ Be aware of the following limits that apply to CloudWatch pipelines +###### Note @@ -56,0 +56 @@ Be aware of the following limits that apply to CloudWatch pipelines +Pipeline definitions are not encrypted and should never include sensitive data, such as personally identifiable information (PII). @@ -58 +58 @@ Be aware of the following limits that apply to CloudWatch pipelines -CloudWatch pipelines capabilities are offered as part of existing CloudWatch Logs Data ingestion pricing at no additional cost with metering occurring at time of ingestion. +###### Note @@ -60 +60 @@ CloudWatch pipelines capabilities are offered as part of existing CloudWatch Log -Existing log class ingestion pricing for vended and custom logs are applicable. For example, if you ingest 30GB of custom standard class logs and process them via a pipeline it would cost $15/Month on logs ingestion. If you ingest and process infrequent access custom logs then you would pay $7.50/Month. Please refer to [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/) for more details on Log Data Ingestion Pricing. +Adding processors leads to mutation of the log events and original (raw) logs are not retained. @@ -62 +62 @@ Existing log class ingestion pricing for vended and custom logs are applicable. -Ingestion of data from 3P connector sources or via an S3 bucket connector is classified as Custom logs ingestion and follows Custom Log Data Ingestion pricing. Metering occurs at time of data on ingestion into CloudWatch. 3P connector sources or via an S3 bucket connector are metered after processing. CloudWatch pipeline sources metering including Vended and custom CloudWatch logs sources occurs at time logs are first received which is before processing via a pipeline. +## Pricing @@ -64 +64 @@ Ingestion of data from 3P connector sources or via an S3 bucket connector is cla -CloudWatch pipelines is available in the following AWS Regions. +CloudWatch pipelines is included with CloudWatch Logs at no additional cost. Standard log ingestion rates based on log class (vended or custom) and storage class (Standard or Infrequent Access) still apply. Metering occurs at time of first ingestion into CloudWatch. CloudWatch Logs sources are metered before pipeline processing. Third-party and S3 bucket sources are classified as Custom logs and metered after processing. For pricing details, see [CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). @@ -66 +66 @@ CloudWatch pipelines is available in the following AWS Regions. -###### Note +## Region availability @@ -68 +68 @@ CloudWatch pipelines is available in the following AWS Regions. -Third-party data source collection is available in regions where [OpenSearch Ingestion has API endpoints](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html). +CloudWatch pipelines is available in the following AWS Regions: @@ -115,15 +115 @@ Third-party data source collection is available in regions where [OpenSearch Ing -For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cw_region.html) in the _AWS General Reference_. - -###### Topics - - * [Creating pipelines](./Creating-pipelines.html) - - * [Managing pipelines](./managing-pipelines.html) - - * [Data sources](./data-sources.html) - - * [CloudWatch pipelines processors](./pipeline-processors.html) - - * [Sinks](./pipeline-sinks.html) - - * [CloudWatch pipelines IAM policies and permissions](./pipeline-iam-reference.html) +###### Note @@ -131 +117 @@ For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws. - * [CloudWatch pipelines extensions](./pipeline-extensions.html) +Third-party data source collection is available in regions where [OpenSearch Ingestion has API endpoints](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html). @@ -133 +119 @@ For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws. - * [Monitoring Pipelines Using CloudWatch Metrics](./pipelines-metrics.html) +For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cw_region.html) in the _AWS General Reference_. @@ -135 +121 @@ For more details, see [Amazon CloudWatch endpoints and quotas](https://docs.aws. - * [Troubleshooting](./troubleshooting.html) +###### Topics