AWS IDR documentation change
Summary
Restructured and expanded the testing documentation for AWS Incident Detection and Response. Added detailed testing options (Scheduled GameDay and Offline testing), specific CLI commands for CloudWatch alarm testing, third-party APM testing procedures, key outcomes, and an FAQ section.
Security assessment
The changes focus on improving testing procedures and documentation clarity. No security vulnerabilities, weaknesses, or incidents are mentioned. The added content emphasizes operational best practices like disabling alarm actions during testing to prevent unintended actions, but this is not security-specific documentation.
Diff
diff --git a/IDR/latest/userguide/idr-workloads-testing.md b/IDR/latest/userguide/idr-workloads-testing.md index 13f89e6b7..a2ded8786 100644 --- a//IDR/latest/userguide/idr-workloads-testing.md +++ b//IDR/latest/userguide/idr-workloads-testing.md @@ -7 +7 @@ -CloudWatch alarmsThird party APM alarmsKey outputs +Testing optionsHow to test your alarmsKey outcomesFAQ @@ -10,0 +11,65 @@ CloudWatch alarmsThird party APM alarmsKey outputs +After [Alarm Ingestion](./idr-gs-alarm-ingestion.html) completes, AWS Incident Detection and Response enables monitoring for your workload and sends a Go-Live confirmation. Your workload is actively monitored from this point forward. + +Alarm testing validates that your onboarded alarms engage AWS Incident Detection and Response as expected, trigger the appropriate runbooks, and any other desired actions, such as auto case creation if you selected it during alarm ingestion. + +Testing is optional but strongly recommended. You're responsible for validating your response arrangements before a real incident occurs. + +## Testing options + +AWS Incident Detection and Response offers two testing options. + +### Option 1: Scheduled GameDay (recommended) + +A scheduled GameDay is a live end-to-end simulation of what might happen during a real incident. AWS Incident Detection and Response follows your prescribed [runbook](./idr-workloads-dev-runbook.html) steps to give you insight into how a real incident might unfold. The GameDay is an opportunity for you to ask questions or refine instructions to improve the engagement. + +###### To schedule a GameDay, complete the following steps: + + 1. [Notify AWS Incident Detection and Response](./idr-workloads-change-request.html) with a preferred date and a 1-hour time window, including time zone. Provide at least 48 hours of lead time. + + 2. Plan resources for the GameDay, including your SRE/Ops team and escalation contacts. + + + + +**GameDay schedule:** + + 1. You and AWS Incident Detection and Response join the call. + + 2. You disable alarm actions, if applicable. + + 3. You manually set your alarms to the **ALARM** state using the instructions in How to test your alarms. + + 4. AWS Incident Detection and Response confirms receipt of the alarm notification. + + 5. AWS Incident Detection and Response responds to the alarm and joins the bridge prescribed in your runbook. + + 6. You and AWS Incident Detection and Response confirm the GameDay outcome. + + + + +### Option 2: Offline alarm testing + +You can test your alarms independently at any time without scheduling a call. Triggering an alarm engages AWS Incident Detection and Response according to your runbook, just as it would during a real incident. + +###### To perform offline alarm testing, complete the following steps: + + 1. To prevent unintended actions, disable any Amazon CloudWatch alarm actions. + + 2. Trigger your alarms using the instructions in How to test your alarms. + + 3. Within 5 minutes, a support case is created on your behalf and AWS Incident Detection and Response engages you as specified in your runbook. + + 4. Notify the Incident Manager that you are conducting offline alarm testing. + + 5. The Incident Manager confirms which alarm state changes were received and validates the response arrangements. + + + + +If a support case isn't created within 5 minutes, submit an [incident request](./inbound-incident-idr.html) to manually engage AWS Incident Detection and Response for troubleshooting. + +## How to test your alarms + +### Amazon CloudWatch alarms + @@ -15 +80 @@ The AWS Identity and Access Management user or role that you use for alarm testi -The last step in the onboarding process is to perform a gameday for your new workload. After alarm ingestion completes, AWS Incident Detection and Response confirms a date and time of your choosing to start your gameday. +Use the AWS Command Line Interface or [AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html) to manually set your alarm to the **ALARM** state. These commands change the alarm state without impacting your workload. @@ -17 +82 @@ The last step in the onboarding process is to perform a gameday for your new wor -Your gameday serves two main purposes: +To prevent unintended actions, for example Amazon EC2 instance restarts, disable any CloudWatch alarm actions before you change the alarm state. You can re-enable CloudWatch alarm actions after testing completes. To learn more about disabling or enabling alarm actions, see [DisableAlarmActions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DisableAlarmActions.html) and [EnableAlarmActions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_EnableAlarmActions.html) in the _Amazon CloudWatch API Reference_. @@ -19 +84 @@ Your gameday serves two main purposes: - * **Functional Validation:** Confirms that AWS Incident Detection and Response can correctly receive your alarm events. And, functional validation confirms that your alarm events trigger the appropriate runbooks and any other desired actions, such as auto case creation if you selected it during alarm ingestion. +**Disable alarm actions:** @@ -21 +85,0 @@ Your gameday serves two main purposes: - * **Simulation:** The gameday is an end to end simulation of what might happen during a real incident. AWS Incident Detection and Response follows your prescribed runbook steps to give you insight into how a real incident might unfold. The gameday is an opportunity for you to ask questions or refine instructions to improve the engagement. @@ -22,0 +87 @@ Your gameday serves two main purposes: + aws cloudwatch disable-alarm-actions --alarm-names "ExampleAlarm" --region us-east-1 @@ -23,0 +89 @@ Your gameday serves two main purposes: +**Set alarm state to ALARM:** @@ -26 +92 @@ Your gameday serves two main purposes: -During the alarm test, AWS Incident Detection and Response works with you to remediate any issues identified. + aws cloudwatch set-alarm-state --alarm-name "ExampleAlarm" --state-value ALARM --state-reason "Testing AWS Incident Detection and Response" --region us-east-1 @@ -28 +94 @@ During the alarm test, AWS Incident Detection and Response works with you to rem -## CloudWatch alarms +**Re-enable alarm actions after testing:** @@ -30 +95,0 @@ During the alarm test, AWS Incident Detection and Response works with you to rem -AWS Incident Detection and Response tests your Amazon CloudWatch alarms by monitoring the state change of your alarm. To do this, manually change the alarm to the **Alarm** state using the AWS Command Line Interface. You can also access the AWS CLI from AWS CloudShell. AWS Incident Detection and Response provides you with a list of AWS CLI commands for you to use during testing. @@ -32 +97 @@ AWS Incident Detection and Response tests your Amazon CloudWatch alarms by monit -To prevent unwanted actions, for example Amazon EC2 instance restarts, disable any CloudWatch alarm actions before you change the alarm state. You can re-enable CloudWatch alarm actions after the testing completes. To learn more about disabling or enabling alarm actions, see [DisableAlarmActions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DisableAlarmActions.html) and [EnableAlarmActions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_EnableAlarmActions.html) in the _Amazon CloudWatch API Reference_. + aws cloudwatch enable-alarm-actions --alarm-names "ExampleAlarm" --region us-east-1 @@ -34 +99 @@ To prevent unwanted actions, for example Amazon EC2 instance restarts, disable a -Example AWS CLI command to set an alarm state: +The alarm state reverts to **OK** automatically within a few seconds. @@ -35,0 +101 @@ Example AWS CLI command to set an alarm state: +**Composite alarms** @@ -37 +103 @@ Example AWS CLI command to set an alarm state: - aws cloudwatch set-alarm-state --alarm-name "ExampleAlarm" --state-value ALARM --state-reason "Testing AWS Incident Detection and Response" --region us-east-1 +The `set-alarm-state` command doesn't guarantee that composite alarms revert to the **OK** state. As a best practice, verify the state of composite alarms after testing. To manually reset a composite alarm, use the following command: @@ -39 +105,4 @@ Example AWS CLI command to set an alarm state: -To learn more about manually changing the state of CloudWatch alarms, see [SetAlarmState](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_SetAlarmState.html). + + aws cloudwatch set-alarm-state --alarm-name "ExampleCompositeAlarm" --state-value OK --state-reason "Testing AWS Incident Detection and Response" --region us-east-1 + +To learn more about manually changing the state of CloudWatch alarms, see [SetAlarmState](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_SetAlarmState.html) in the _Amazon CloudWatch API Reference_. @@ -43 +112,36 @@ To learn more about the permissions required for CloudWatch API operations, see -## Third party APM alarms +### Third-party APM alarms + +Workloads that use a third-party Application Performance Monitoring (APM) tool, such as Datadog, Splunk, New Relic, or Dynatrace, require different instructions to simulate an alarm. + + 1. Disable alarm actions in your APM to prevent unintended actions. + + 2. Modify your alarm threshold or comparison operator to force the alarm into the **ALARM** status. This triggers a payload to AWS Incident Detection and Response. + + 3. After testing completes, roll back the threshold or comparison operator changes to restore the alarm to **OK** status. + + + + +## Key outcomes + +After successful testing: + + * Alarm ingestion is confirmed and your alarm configuration is correct. + + * Alarms are received by AWS Incident Detection and Response. + + * A support case is created and your prescribed contacts are notified. + + * AWS Incident Detection and Response engages you by your prescribed conference means. + + * All alarms and support cases generated during testing are resolved. + + + + +## Frequently asked questions + +**Is alarm testing mandatory?** + + +No. Testing is optional but strongly recommended to validate your end-to-end response arrangements before a real incident occurs. @@ -45 +149 @@ To learn more about the permissions required for CloudWatch API operations, see -Workloads that utilize a third party Application Performance Monitoring (APM) tool, such as Datadog, Splunk, New Relic, or Dynatrace, require different instructions to simulate an alarm. At the start of the gameday, AWS Incident Detection and Response requests that you temporarily change your alarm thresholds or comparison operators to force the alarm into the **ALARM** status. This status triggers a payload to AWS Incident Detection and Response. +**Will my workload be impacted?** @@ -47 +150,0 @@ Workloads that utilize a third party Application Performance Monitoring (APM) to -## Key outputs @@ -49 +152 @@ Workloads that utilize a third party Application Performance Monitoring (APM) to -Key outputs: +No. However, during testing any alarm actions configured on your alarms are triggered unless you disable them. Disable alarm actions before testing to prevent unintended impacts. @@ -51 +154 @@ Key outputs: - * Alarm ingestion is successful and your alarm configuration is correct. +**Who is notified during testing?** @@ -53 +155,0 @@ Key outputs: - * Alarms are successfully created and received by AWS Incident Detection and Response. @@ -55 +157 @@ Key outputs: - * A support case is created for your engagement and your prescribed contacts are notified. +During a scheduled GameDay, all contacts and escalation paths in your runbook are contacted for verification. During offline alarm testing, only the initial contact specified during alarm onboarding is notified. @@ -57 +159 @@ Key outputs: - * AWS Incident Detection and Response can engage with you by your prescribed conference means. +**Can I reply via email to case updates?** @@ -59 +160,0 @@ Key outputs: - * All alarms and support cases generated as part of the gameday are resolved. @@ -61 +162 @@ Key outputs: - * A Go-Live email is sent confirming your workload is now being monitored by AWS Incident Detection and Response. +No. Email copies of Support case correspondences are sent from a no-reply address. To update a case, use the [AWS Support Center Console](https://console.aws.amazon.com/support/home#/). @@ -62,0 +164 @@ Key outputs: +**How do I request a GameDay after go-live?** @@ -64,0 +167 @@ Key outputs: +Reply to your existing onboarding support case, if it exists, or create a [Request changes to an onboarded workload in Incident Detection and Response](./idr-workloads-change-request.html).