AWS deadline-cloud documentation change
Summary
Added detailed monitoring documentation for host configuration script execution including CloudWatch Logs, Insights, and metric filters
Security assessment
The change adds monitoring capabilities for script execution failures but does not address a specific security vulnerability. Enhances security posture by documenting how to detect configuration failures which could indicate security misconfigurations.
Diff
diff --git a/deadline-cloud/latest/developerguide/smf-admin.md b/deadline-cloud/latest/developerguide/smf-admin.md index 957e19bad..cd21eb9ac 100644 --- a//deadline-cloud/latest/developerguide/smf-admin.md +++ b//deadline-cloud/latest/developerguide/smf-admin.md @@ -140 +140 @@ To monitor the script: - 1. Open the fleet page in the Deadline Cloud console + 1. Open the fleet page in the Deadline Cloud console. @@ -142 +142 @@ To monitor the script: - 2. Choose **View workers** to open the Deadline Cloud monitor + 2. Choose **View workers** to open the Deadline Cloud monitor. @@ -144 +144 @@ To monitor the script: - 3. View the worker status in the monitor page + 3. View the worker status in the monitor page. @@ -163,0 +164,80 @@ Within that log group is a stream of +### Monitoring host configuration script execution + +With Admin scripts to configure workers, you can take full control of a Deadline Cloud worker. You can install any software package, reconfigure operating system parameters, or mount shared file systems. With this advanced feature and Deadline Cloud's capability to scale to thousands of workers, you can now monitor when configuration scripts are executed successfully or failed. Failure causes may include script errors, flaky behavior, or other unknown situations. + +We recommend the following solutions for monitoring host configuration script execution. + +#### CloudWatch Logs monitoring + +All fleet host configuration logs are streamed to the fleet's CloudWatch log group, and specifically to a worker's CloudWatch log stream. For example, `/aws/deadline/farm-123456789012/fleet-777788889999` is the log group for farm `123456789012`, fleet `777788889999`. + +Each worker provisions a dedicated log stream, for example `worker-123456789012`. Host configuration logs include log banners such as _Running Host Configuration Script_ and _Finished running Host Configuration Script, exit code: 0_. The exit code of the script is included in the finished banner and can be queried using CloudWatch tools. + +#### CloudWatch Logs Insights + +CloudWatch Logs Insights offers advanced capabilities to analyze log information. For example, the following Log Insights query parses for the host configuration exit code, sorted by time: + + + fields @timestamp, @message, @logStream, @log + | filter @message like /Finished running Host Configuration Script/ + | parse @message /exit code: (?<exit_code>\d+)/ + | display @timestamp, exit_code + | sort @timestamp desc + +For more information about CloudWatch Logs Insights, see [Analyzing log data with CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) in the _Amazon CloudWatch Logs User Guide_. + +##### Worker agent structured logging + +Deadline Cloud's worker agent publishes structured JSON logs to CloudWatch. The worker agent offers a wide array of structured logs for analyzing worker health. For more information, see [Deadline Cloud worker agent logging](https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/docs/logging.md) on GitHub. + +The attributes of the structured logs are unpacked to fields in Log Insights. You can use this CloudWatch capability to count and analyze host configuration startup failures. For example, a count and bin query can be used to determine how often failures occur: + + + fields @timestamp, @message, @logStream, @log + | sort @timestamp desc + | filter message like /Worker Agent host configuration failed with exit code/ + | stats count(*) by exit_code, bin(1h) + +#### CloudWatch metric filters for metrics and alarming + +You can set up CloudWatch metric filters to generate CloudWatch metrics from logs. This lets you create alarms and dashboards for monitoring host configuration script execution. + +###### To create a metric filter + + 1. Open the CloudWatch console. + + 2. In the navigation pane, choose **Logs** , then **Log groups**. + + 3. Select your fleet's log group. + + 4. Choose **Create metric filter**. + + 5. Define your filter pattern using one of the following: + + * **For success metrics:** + + {$.message = "*Worker Agent host configuration succeeded.*"} + + * **For failure metrics:** + + {$.exit_code != 0 && $.message = "*Worker Agent host configuration failed with exit code*"} + + 6. Choose **Next** to create a metric with the following values: + + * **Metric namespace:** Your metric namespace (for example, `MyDeadlineFarm`) + + * **Metric name:** Your requested metric name (for example, `host_config_failure`) + + * **Metric value:** `1` (each instance is a count of 1) + + * **Default value:** Leave empty + + * **Unit:** `Count` + + + + +After creating metric filters, you can configure standard CloudWatch alarms to take action on elevated host configuration failure rates, or add the metrics to a CloudWatch dashboard for day-to-day operations and monitoring. + +For more details, see [Filter and pattern syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) in the _Amazon CloudWatch Logs User Guide_. +