AWS pcs documentation change
Summary
Expanded scheduler logging documentation to include slurmdbd and slurmrestd daemons, updated version numbers, and added example log records for new daemon types
Security assessment
This change enhances monitoring capabilities by adding logging for additional Slurm daemons (slurmdbd and slurmrestd). While improved logging supports security auditing and monitoring, there's no evidence this addresses a specific security vulnerability. The change expands existing security documentation by providing more comprehensive logging coverage.
Diff
diff --git a/pcs/latest/userguide/monitoring_scheduler-logs.md b/pcs/latest/userguide/monitoring_scheduler-logs.md index e69238a6a..e1a16bc64 100644 --- a//pcs/latest/userguide/monitoring_scheduler-logs.md +++ b//pcs/latest/userguide/monitoring_scheduler-logs.md @@ -7 +7 @@ -PrerequisitesSet up scheduler logsScheduler log stream paths and namesExample scheduler log record +PrerequisitesSet up scheduler logsScheduler log stream paths and namesExample scheduler log records @@ -12,0 +13,13 @@ You can configure AWS PCS to send detailed logging data from your cluster schedu +AWS PCS delivers logs from the following Slurm daemons through the `PCS_SCHEDULER_LOGS` log type: + + * **`slurmctld`** — The Slurm controller daemon. Available for all supported Slurm versions. + + * **`slurmdbd`** — The Slurm database daemon. Available for Slurm 24.11 and later. + + * **`slurmrestd`** — The Slurm REST API daemon. Available for Slurm 25.05 and later. + + + + +Clusters that already have `PCS_SCHEDULER_LOGS` delivery configured automatically start receiving `slurmdbd` and `slurmrestd` logs when they run a supported Slurm version. No additional configuration is required. + @@ -21 +34 @@ You can configure AWS PCS to send detailed logging data from your cluster schedu - * [Example scheduler log record](./monitoring_scheduler-logs.html#monitoring_scheduler-logs_record) + * [Example scheduler log records](./monitoring_scheduler-logs.html#monitoring_scheduler-logs_record) @@ -139,0 +153,2 @@ The path and name for AWS PCS scheduler logs depend on the destination type. +The `${log_name}` value in the paths below is `slurmctld`, `slurmdbd`, or `slurmrestd`, depending on the daemon that produced the log. + @@ -148 +163,3 @@ The path and name for AWS PCS scheduler logs depend on the destination type. - AWSLogs/PCS/abcdef0123/slurmctld_24.05.log + AWSLogs/PCS/abcdef0123/slurmctld_25.11.log + AWSLogs/PCS/abcdef0123/slurmdbd_24.11.log + AWSLogs/PCS/abcdef0123/slurmrestd_25.05.log @@ -158 +175,3 @@ The path and name for AWS PCS scheduler logs depend on the destination type. - AWSLogs/111111111111/PCS/us-east-2/abcdef0123/slurmctld/24.05/2024/09/01/00. + AWSLogs/111111111111/PCS/us-east-2/abcdef0123/slurmctld/25.11/2024/09/01/00/ + AWSLogs/111111111111/PCS/us-east-2/abcdef0123/slurmdbd/24.11/2024/09/01/00/ + AWSLogs/111111111111/PCS/us-east-2/abcdef0123/slurmrestd/25.05/2024/09/01/00/ @@ -166 +185,2 @@ The path and name for AWS PCS scheduler logs depend on the destination type. - PCS_slurmctld_24.05_2024-09-01-00_abcdef0123_0123abcdef.log + PCS_slurmctld_25.11_2024-09-01-00_abcdef0123_0123abcdef.log + @@ -169,0 +190 @@ The path and name for AWS PCS scheduler logs depend on the destination type. +## Example scheduler log records @@ -171 +192 @@ The path and name for AWS PCS scheduler logs depend on the destination type. -## Example scheduler log record +AWS PCS scheduler logs are structured. They include fields such as the cluster identifier, scheduler type, major and patch versions, in addition to the log message emitted from the Slurm daemon process. The `log_name` and `node_type` fields identify which daemon produced the log. @@ -173 +194 @@ The path and name for AWS PCS scheduler logs depend on the destination type. -AWS PCS scheduler logs are structured. They include fields such as the cluster identifier, scheduler type, major and patch versions, in addition to the log message emitted from the Slurm controller process. Here is an example. +The following example shows a `slurmctld` log record. @@ -183,2 +204,2 @@ AWS PCS scheduler logs are structured. They include fields such as the cluster i - "scheduler_major_version": "25.05", - "scheduler_patch_version": "3", + "scheduler_major_version": "25.11", + "scheduler_patch_version": "2", @@ -188,0 +210,32 @@ AWS PCS scheduler logs are structured. They include fields such as the cluster i +The following example shows a `slurmdbd` log record (Slurm 24.11 and later). + + + { + "resource_id": "pcs_bu93qsds2j", + "resource_type": "PCS_CLUSTER", + "event_timestamp": 1774485082772, + "log_level": "info", + "log_name": "slurmdbd", + "scheduler_type": "slurm", + "scheduler_major_version": "25.11", + "scheduler_patch_version": "2", + "node_type": "slurmdbd_primary", + "message": "[2026-03-26T00:31:22.772+00:00] mysql_common: storage token refreshed" + } + +The following example shows a `slurmrestd` log record (Slurm 25.05 and later). + + + { + "resource_id": "pcs_bu93qsds2j", + "resource_type": "PCS_CLUSTER", + "event_timestamp": 1774485082772, + "log_level": "info", + "log_name": "slurmrestd", + "scheduler_type": "slurm", + "scheduler_major_version": "25.05", + "scheduler_patch_version": "3", + "node_type": "slurmrestd_primary", + "message": "[2026-03-26T00:31:22.772+00:00] slurmrestd: Listening on port 6820\n" + } + @@ -197 +250 @@ Job completion logs -Monitoring with CloudWatch +Scheduler audit logs