AWS wickr medium security documentation change
Summary
Major expansion of troubleshooting guide adding sections on bot runtime architecture, monitoring with CloudWatch, log collection, common issues, and improving bot resilience
Security assessment
The changes include specific security warnings about debug logging potentially capturing sensitive authentication challenge tokens, guidance on preventing disk exhaustion from unrotated Docker logs, and instructions for securing bot connections through proper firewall/security group configurations. The documentation explicitly warns about security risks and provides mitigation strategies.
Diff
diff --git a/wickr/latest/wickrio/troubleshooting.md b/wickr/latest/wickrio/troubleshooting.md index 0e6db292d..18f5d7350 100644 --- a//wickr/latest/wickrio/troubleshooting.md +++ b//wickr/latest/wickrio/troubleshooting.md @@ -7 +7 @@ -Setting up Wickr IO Docker containerProvisioning Wickr IO clientStart bot client failuresWickr IO command line interfaceClient and Integration compatibility issuesDeploying custom IntegrationsOther issuesUpgrading bots +Bot runtime architectureSetting up Wickr IO Docker containerProvisioning Wickr IO clientStart bot client failuresWickr IO command line interfaceClient and Integration compatibility issuesDeploying custom IntegrationsMonitor bot health with Amazon CloudWatchCollect bot logsCommon issuesUpgrading botsImprove bot resilience @@ -14,0 +15,2 @@ This guide provides documentation for Wickr IO Integrations. If you're using AWS + * Bot runtime architecture + @@ -27 +29,5 @@ This guide provides documentation for Wickr IO Integrations. If you're using AWS - * Other issues + * Monitor bot health with Amazon CloudWatch + + * Collect bot logs + + * Common issues @@ -30,0 +37,2 @@ This guide provides documentation for Wickr IO Integrations. If you're using AWS + * Improve bot resilience + @@ -37,0 +46,19 @@ This section will describe some possible issues you may run into while using the +###### Note + +AWS Support can help with AWS Wickr service configuration, network settings, and the Wickr bot SDK APIs. For issues with custom bot logic, container infrastructure, or host environment, contact your internal development or operations team. + +## Bot runtime architecture + +Each Wickr IO bot has three layers, which AWS recommends independent monitoring for: + + 1. **Host** – The Amazon EC2 instance or on-premises server running Docker. + + 2. **Container** – The Wickr IO Docker container (`wickrio/bot-cloud` or `wickrio/bot-cloud-govcloud`). + + 3. **Bot process** – Your Node.js integration running inside the container. + + + + +The container can report as running while the bot process inside it has crashed or lost its connection to the Wickr network. Effective monitoring covers all three layers. + @@ -234 +261,332 @@ Make sure all the required files are present in the zipped integration software -## Other issues +## Monitor bot health with Amazon CloudWatch + +The following table summarizes the monitoring approaches for each layer, from simplest to most comprehensive. + +Layer | What it catches | Effort | Reliability +---|---|---|--- +Amazon EC2 status check | Host failure, hardware issues | None (built-in) | High +Container metric | Docker crash, OOM kill, restart loop | Low | Medium +Log-based metric filter | Container crash, restart loop (container output only) | Low | Low +Heartbeat metric | All of the above, plus silent disconnects and event loop blocks | Medium | High + +For production bots, we recommend implementing all four approaches. They are not redundant – each catches different failure modes. + +### Monitor the host instance + +Use the built-in Amazon EC2 `StatusCheckFailed` metric to detect host-level failures. On supported instance types, Amazon EC2 simplified automatic recovery is enabled by default and automatically migrates the instance to healthy hardware when a status check fails. The alarm below is informational – it notifies you that a recovery occurred, but no manual action is required. For more information, see [Recover your instance](https://docs.aws.amazon.com//AWSEC2/latest/UserGuide/ec2-instance-recover.html) in the _Amazon EC2 User Guide_. + + + HostDownAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: !Sub "${BotName}-HostDown" + Namespace: AWS/EC2 + MetricName: StatusCheckFailed + Dimensions: + - Name: InstanceId + Value: !Ref BotInstance + Statistic: Maximum + Period: 60 + EvaluationPeriods: 2 + Threshold: 1 + ComparisonOperator: GreaterThanOrEqualToThreshold + AlarmActions: + - !Ref AlertTopic + +### Monitor the Docker container + +Create a script on the host that publishes a custom CloudWatch metric based on container status. Run it every minute using `cron`. For example: + + + #!/bin/bash + # /opt/wickr-bot/monitor.sh + CONTAINER_NAME="${1:-wickr-bot}" + NAMESPACE="WickrIO/Bots" + + STATUS=$(docker inspect --format='{{.State.Running}}' "$CONTAINER_NAME" 2>/dev/null) + + aws cloudwatch put-metric-data \ + --namespace "$NAMESPACE" \ + --metric-name ContainerRunning \ + --dimensions BotName="$CONTAINER_NAME" \ + --value "$([ "$STATUS" = "true" ] && echo 1 || echo 0)" \ + --unit Count + +Add a cron entry to run the script: + + + * * * * * /opt/wickr-bot/monitor.sh container-name + +Create an alarm that fires when the value drops to 0. + +### Monitor with log-based metric filters + +If you forward container logs to Amazon CloudWatch Logs using the `awslogs` driver, you can create a metric filter that counts log events. When the container stops producing output, the alarm fires. + + 1. Open the Amazon CloudWatch console and navigate to **Log groups**. + + 2. Select your bot's log group (`/wickr/bots/`bot-name``). + + 3. Choose **Metric filters** , then **Create metric filter**. + + 4. For **Filter pattern** , enter a single space (`" "`) to match any log line. + + 5. Set the metric namespace to `WickrIO/Bots` and the metric name to `BotLogEvents`. + + 6. Create an alarm on this metric: `Sum < 1` over a 5-minute period. + + + + +### Add a heartbeat metric (recommended) + +A heartbeat metric is the most reliable approach because it proves the Node.js process is alive and the event loop is not blocked. Add the following to your bot's entry point, after the bot starts successfully. For example: + + + const { CloudWatchClient, PutMetricDataCommand } = require('@aws-sdk/client-cloudwatch') + const cw = new CloudWatchClient() + const BOT_NAME = process.env.BOT_NAME || 'wickr-bot' + + setInterval(async () => { + try { + await cw.send(new PutMetricDataCommand({ + Namespace: 'WickrIO/Bots', + MetricData: [{ + MetricName: 'Heartbeat', + Dimensions: [{ Name: 'BotName', Value: BOT_NAME }], + Value: 1, + Unit: 'Count' + }] + })) + } catch (e) { console.error('Heartbeat publish failed:', e.message) } + }, 60000) + +Add the `@aws-sdk/client-cloudwatch` dependency to your bot's `package.json`. + +Create an alarm on the heartbeat metric. For example: + + + BotProcessAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: !Sub "${BotName}-ProcessDown" + Namespace: WickrIO/Bots + MetricName: Heartbeat + Dimensions: + - Name: BotName + Value: !Ref BotName + Statistic: Sum + Period: 300 + EvaluationPeriods: 2 + Threshold: 1 + ComparisonOperator: LessThanThreshold + TreatMissingData: breaching + AlarmActions: + - !Ref AlertTopic + +Set `TreatMissingData` to `breaching` so the alarm fires when the metric stops arriving entirely, rather than entering `INSUFFICIENT_DATA`. + +### Monitor memory usage + +Node.js bots that maintain session state, cache data, or process file attachments can develop memory leaks over time. If the bot's memory usage exceeds the Docker container's memory limit, Docker terminates the container with an out-of-memory (OOM) kill. The container restarts automatically if you configured the `--restart` policy, but active user sessions and in-flight messages are lost. + +To detect memory growth before it causes an OOM kill, publish the Node.js heap usage alongside your heartbeat metric. For example: + + + setInterval(async () => { + const mem = process.memoryUsage() + try { + await cw.send(new PutMetricDataCommand({ + Namespace: 'WickrIO/Bots', + MetricData: [ + { + MetricName: 'Heartbeat', + Dimensions: [{ Name: 'BotName', Value: BOT_NAME }], + Value: 1, Unit: 'Count' + }, + { + MetricName: 'HeapUsedMB', + Dimensions: [{ Name: 'BotName', Value: BOT_NAME }], + Value: Math.round(mem.heapUsed / 1048576), + Unit: 'Megabytes' + } + ] + })) + } catch (e) { console.error('Metric publish failed:', e.message) } + }, 60000) + +Create a CloudWatch alarm that fires when heap usage exceeds 80% of the container's memory limit. For example, if the container has a 512 MB memory limit (`--memory=512m`), set the threshold to 410 MB. Adjust the threshold to 80% of your configured limit.