AWS fsx documentation change
Summary
Removed documentation section about using HSM commands and validation script for checking released objects
Security assessment
Change removes operational documentation about HSM file management but shows no evidence of addressing security vulnerabilities or weaknesses
Diff
diff --git a/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.md b/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.md index 807bc6120..5f2f02fb0 100644 --- a/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.md +++ b/fsx/latest/LustreGuide/preload-file-contents-hsm-dra.md @@ -5,2 +4,0 @@ -Importing files using HSM commandsValidation step - @@ -9,4 +6,0 @@ Importing files using HSM commandsValidation step -You can optionally preload contents individual files or directories into your file system. - -## Importing files using HSM commands - @@ -30,75 +23,0 @@ If your linked S3 bucket is larger than your file system, you should be able to -## Validation step - -You can run the bash script listed below to help you discover how many files or objects are in an archived (released) state. - -To improve the script's performance, especially across file systems with a large number of files, CPU threads are automatically determined based in the `/proc/cpuproc` file. That is, you will see faster performance with a higher vCPU count Amazon EC2 instance. - - 1. Set up the bash script. - - #!/bin/bash - - # Check if a directory argument is provided - if [ $# -ne 1 ]; then - echo "Usage: $0 /path/to/lustre/mount" - exit 1 - fi - - # Set the root directory from the argument - ROOT_DIR="$1" - - # Check if the provided directory exists - if [ ! -d "$ROOT_DIR" ]; then - echo "Error: Directory $ROOT_DIR does not exist." - exit 1 - fi - - # Automatically detect number of CPUs and set threads - if command -v nproc &> /dev/null; then - THREADS=$(nproc) - elif [ -f /proc/cpuinfo ]; then - THREADS=$(grep -c ^processor /proc/cpuinfo) - else - echo "Unable to determine number of CPUs. Defaulting to 1 thread." - THREADS=1 - fi - - # Output file - OUTPUT_FILE="released_objects_$(date +%Y%m%d_%H%M%S).txt" - - echo "Searching in $ROOT_DIR for all released objects using $THREADS threads" - echo "This may take a while depending on the size of the filesystem..." - - # Find all released files in the specified lustre directory using parallel - time sudo lfs find "$ROOT_DIR" -type f | \ - parallel --will-cite -j "$THREADS" -n 1000 "sudo lfs hsm_state {} | grep released" > "$OUTPUT_FILE" - - echo "Search complete. Released objects are listed in $OUTPUT_FILE" - echo "Total number of released objects: $(wc -l <"$OUTPUT_FILE")" - - 2. Make the script executable: - - $ chmod +x find_lustre_released_files.sh - - 3. Run the script, as in the following example: - - $ ./find_lustre_released_files.sh /fsxl/sample - Searching in /fsxl/sample for all released objects using 16 threads - This may take a while depending on the size of the filesystem... - real 0m9.906s - user 0m1.502s - sys 0m5.653s - Search complete. Released objects are listed in released_objects_20241121_184537.txt - Total number of released objects: 30000 - - - - - -If there are released objects present, then perform a bulk restore on the desired directories to bring the files into FSx for Lustre from S3, as in the following example: - - - $ DIR=/path/to/lustre/mount - $ nohup find $DIR -type f -print0 | xargs -0 -n 1 -P 8 sudo lfs hsm_restore & - -Note that `hsm_restore` will take a while where there are millions of files. -