AWS solutions high security documentation change
Summary
Updated container image to use Amazon Linux 2023, simplified package installation, removed explicit vulnerability patches, added test type handling, and enhanced logging/scripting.
Security assessment
The changes remove explicit steps that addressed specific vulnerabilities (CVE-2024-23334, CVE-2024-3651) by upgrading libraries like aiohttp and idna. The removal of other security-related package upgrades (urllib3, Werkzeug, cryptography) may expose the system to known security issues if the new base image or packages do not include patched versions. This introduces potential security regressions.
Diff
diff --git a/solutions/latest/distributed-load-testing-on-aws/container-image.md b/solutions/latest/distributed-load-testing-on-aws/container-image.md index 58b0beda4..83dc20716 100644 --- a//solutions/latest/distributed-load-testing-on-aws/container-image.md +++ b//solutions/latest/distributed-load-testing-on-aws/container-image.md @@ -7 +7 @@ -This solution uses a public Amazon Elastic Container Registry (Amazon ECR) image repository managed by AWS to store the Taurus image that is used to run the configured tests. If you want to customize the container image, you can rebuild and push the image into an ECR image repository in your own AWS account. +This solution uses a public Amazon Elastic Container Registry (Amazon ECR) image repository managed by AWS to store the image that is used to run the configured tests. If you want to customize the container image, you can rebuild and push the image into an ECR image repository in your own AWS account. @@ -41,5 +41 @@ The following example shows the container file. - FROM blazemeter/taurus:1.16.34 - # taurus includes python and pip - RUN /usr/bin/python3 -m pip install --upgrade pip - RUN pip install --no-cache-dir awscli - RUN apt-get -y install --no-install-recommends xmlstarlet bc procps + FROM public.ecr.aws/amazonlinux/amazonlinux:2023-minimal @@ -47,3 +43,9 @@ The following example shows the container file. - # Removing selenium and gatling from our image as they are not supported in DLT - RUN rm -rf /root/.bzt/selenium-taurus - RUN rm -rf /root/.bzt/gatling-taurus + RUN dnf update -y && \ + dnf install -y python3.11 python3.11-pip java-21-amazon-corretto bc procps jq findutils unzip && \ + dnf clean all + + ENV PIP_INSTALL="pip3.11 install --no-cache-dir" + + + # install bzt + RUN $PIP_INSTALL --upgrade bzt awscli setuptools==70.0.0 @@ -51,2 +53,4 @@ The following example shows the container file. - RUN apt-get update && apt-get upgrade -y - # Taurus working directory = /bzt-configs + # install bzt tools + RUN bzt -install-tools -o modules.install-checker.exclude=selenium,gatling,tsung,siege,ab,k6,external-results-loader,locust,junit,testng,rspec,mocha,nunit,xunit,wdio + RUN rm -rf /root/.bzt/selenium-taurus + RUN mkdir /bzt-configs /tmp/artifacts @@ -61,12 +65 @@ The following example shows the container file. - RUN python3 /bzt-configs/jar_updater.py - - # Remove K6 as it is not supported in DLT by default - RUN apt remove -y k6 - - RUN /bin/bash -c "source /etc/profile.d/rbenv.sh && rbenv uninstall --force $(cat /usr/local/rbenv/version)" - RUN rm -rf /usr/local/rbenv - - # Replacing urllib3 with more stable Versions to resolve vulnerabilities - RUN pip install urllib3==2.2.2 - RUN rm -rf /root/.bzt/python-packages/3.10.12/urllib3* - RUN cp -r /usr/local/lib/python3.10/dist-packages/urllib3* /root/.bzt/python-packages/3.10.12/ + RUN python3.11 /bzt-configs/jar_updater.py @@ -74,4 +67,2 @@ The following example shows the container file. - # Replacing Werkzeug with more stable version to resolve vulnerabilities - RUN pip install Werkzeug==3.0.3 - RUN rm -rf /root/.bzt/python-packages/3.10.12/werkzeug* - RUN cp -r /usr/local/lib/python3.10/dist-packages/werkzeug* /root/.bzt/python-packages/3.10.12/ + # Remove jar files from /tmp + RUN rm -rf /tmp/jmeter-plugins-manager-1.7* @@ -79,4 +70,2 @@ The following example shows the container file. - # Replacing cryptography with more stable version to resolve vulnerabilities - RUN pip install cryptography==42.0.6 - RUN rm -rf /root/.bzt/python-packages/3.10.12/cryptography* - RUN cp -r /usr/local/lib/python3.10/dist-packages/cryptography* /root/.bzt/python-packages/3.10.12/ + # Add settings file to capture the output logs from bzt cli + RUN mkdir -p /etc/bzt.d && echo '{"settings": {"artifacts-dir": "/tmp/artifacts"}}' > /etc/bzt.d/90-artifacts-dir.json @@ -84,13 +73 @@ The following example shows the container file. - # Removing dotnet dependencies as NUnit and Xunit is not supported in DLT - RUN rm -rf /usr/share/dotnet - - # Replacing aiohttp with more stable version to resolve CVE-2024-23334 - RUN rm -rf /usr/local/lib/python3.10/dist-packages/aiohttp* - RUN pip install --upgrade aiohttp - - # Replacing idna with more stable version to resolve CVE-2024-3651 - RUN pip install --upgrade idna - RUN rm -rf /root/.bzt/python-packages/3.10.12/idna* - RUN cp -r /usr/local/lib/python3.10/dist-packages/idna* /root/.bzt/python-packages/3.10.12/ - - WORKDIR /bzt-configs/ + WORKDIR /bzt-configs @@ -99 +76 @@ The following example shows the container file. -In addition to a container file, the directory contains the following bash script that downloads the test configuration from Amazon S3 before running the Taurus program. +In addition to a container file, the directory contains the following bash script that downloads the test configuration from Amazon S3 before running the Taurus/Blazemeter program. @@ -115,0 +93,4 @@ In addition to a container file, the directory contains the following bash scrip + cat /proc/self/cgroup + TASK_ID=$(cat /proc/self/cgroup | grep -oE '[a-f0-9]{32}' | head -n 1) + echo $TASK_ID + @@ -128,0 +110,6 @@ In addition to a container file, the directory contains the following bash scrip + # Set the default log file values to jmeter + LOG_FILE="jmeter.log" + OUT_FILE="jmeter.out" + ERR_FILE="jmeter.err" + KPI_EXT="jtl" + @@ -130,0 +118,9 @@ In addition to a container file, the directory contains the following bash scrip + # setting the log file values to the test type + LOG_FILE="${TEST_TYPE}.log" + OUT_FILE="${TEST_TYPE}.out" + ERR_FILE="${TEST_TYPE}.err" + + # set variables based on TEST_TYPE + if [ "$TEST_TYPE" == "jmeter" ]; then + EXT="jmx" + TYPE_NAME="JMeter" @@ -135,0 +132,2 @@ In addition to a container file, the directory contains the following bash scrip + fi + @@ -137 +135 @@ In addition to a container file, the directory contains the following bash scrip - aws s3 cp s3://$S3_BUCKET/public/test-scenarios/$TEST_TYPE/$TEST_ID.jmx ./ --region $MAIN_STACK_REGION + aws s3 cp s3://$S3_BUCKET/public/test-scenarios/$TEST_TYPE/$TEST_ID.$EXT ./ --region $MAIN_STACK_REGION @@ -141,4 +139,7 @@ In addition to a container file, the directory contains the following bash scrip - # only looks for the first jmx file. - JMETER_SCRIPT=`find . -name "*.jmx" |head -n 1` - if [ -z "$JMETER_SCRIPT" ]; then - echo "There is no JMeter script in the zip file." + echo "UNZIPPED" + ls -l + # only looks for the first test script file. + TEST_SCRIPT=`find . -name "*.${EXT}" | head -n 1` + echo $TEST_SCRIPT + if [ -z "$TEST_SCRIPT" ]; then + echo "There is no test script (.${EXT}) in the zip file." @@ -148 +149 @@ In addition to a container file, the directory contains the following bash scrip - sed -i -e "s|$TEST_ID.jmx|$JMETER_SCRIPT|g" test.json + sed -i -e "s|$TEST_ID.$EXT|$TEST_SCRIPT|g" test.json @@ -170 +171 @@ In addition to a container file, the directory contains the following bash scrip - python3 -u $SCRIPT $TIMEOUT & + python3.11 -u $SCRIPT $TIMEOUT & @@ -175 +176,3 @@ In addition to a container file, the directory contains the following bash scrip - python3 -u $SCRIPT $IPNETWORK $IPHOSTS + aws s3 cp s3://$S3_BUCKET/Container_IPs/${TEST_ID}_IPHOSTS_${AWS_REGION}.txt ./ --region $MAIN_STACK_REGION + export IPHOSTS=$(cat ${TEST_ID}_IPHOSTS_${AWS_REGION}.txt) + python3.11 -u $SCRIPT $IPNETWORK $IPHOSTS @@ -186 +190 @@ In addition to a container file, the directory contains the following bash scrip - cat $TEST_ID.jmx |grep filename > results.txt + cat $TEST_ID.$EXT | grep filename > results.txt @@ -188 +192 @@ In addition to a container file, the directory contains the following bash scrip - cat $JMETER_SCRIPT |grep filename > results.txt + cat $TEST_SCRIPT | grep filename > results.txt @@ -189,0 +194,2 @@ In addition to a container file, the directory contains the following bash scrip + + if [ -f results.txt ]; then @@ -197,0 +204 @@ In addition to a container file, the directory contains the following bash scrip + extensions=() @@ -199 +206,16 @@ In addition to a container file, the directory contains the following bash scrip - p="s3://$S3_BUCKET/results/$TEST_ID/JMeter_Result/$PREFIX/$UUID/$f" + ext="${f##*.}" + if [[ ! " ${extensions[@]} " =~ " ${ext} " ]]; then + extensions+=("$ext") + fi + done + + # Find all files in the current folder with the same extensions + all_files=() + for ext in "${extensions[@]}"; do + for f in *."$ext"; do + all_files+=("$f") + done + done + + for f in "${all_files[@]}"; do + p="s3://$S3_BUCKET/results/$TEST_ID/${TYPE_NAME}_Result/$PREFIX/$UUID/$f" @@ -201 +223 @@ In addition to a container file, the directory contains the following bash scrip - p="s3://$S3_BUCKET/results/$TEST_ID/JMeter_Result/$PREFIX/$UUID$f" + p="s3://$S3_BUCKET/results/$TEST_ID/${TYPE_NAME}_Result/$PREFIX/$UUID$f" @@ -207,0 +230 @@ In addition to a container file, the directory contains the following bash scrip + fi @@ -209,0 +233,18 @@ In addition to a container file, the directory contains the following bash scrip + + # Insert the Task ID at the same level as <FinalStatus> + curl -s $ECS_CONTAINER_METADATA_URI_V4/task + Task_CPU=$(curl -s $ECS_CONTAINER_METADATA_URI_V4/task | jq '.Limits.CPU') + Task_Memory=$(curl -s $ECS_CONTAINER_METADATA_URI_V4/task | jq '.Limits.Memory') + START_TIME=$(curl -s "$ECS_CONTAINER_METADATA_URI_V4/task" | jq -r '.Containers[0].StartedAt') + # Convert start time to seconds since epoch + START_TIME_EPOCH=$(date -d "$START_TIME" +%s) + # Calculate elapsed time in seconds + CURRENT_TIME_EPOCH=$(date +%s) + ECS_DURATION=$((CURRENT_TIME_EPOCH - START_TIME_EPOCH)) + + + sed -i.bak 's/<\/FinalStatus>/<TaskId>'"$TASK_ID"'<\/TaskId><\/FinalStatus>/' /tmp/artifacts/results.xml + sed -i 's/<\/FinalStatus>/<TaskCPU>'"$Task_CPU"'<\/TaskCPU><\/FinalStatus>/' /tmp/artifacts/results.xml + sed -i 's/<\/FinalStatus>/<TaskMemory>'"$Task_Memory"'<\/TaskMemory><\/FinalStatus>/' /tmp/artifacts/results.xml + sed -i 's/<\/FinalStatus>/<ECSDuration>'"$ECS_DURATION"'<\/ECSDuration><\/FinalStatus>/' /tmp/artifacts/results.xml + @@ -211 +252 @@ In addition to a container file, the directory contains the following bash scrip - TEST_DURATION=`xmlstarlet sel -t -v "/FinalStatus/TestDuration" /tmp/artifacts/results.xml` + TEST_DURATION=$(grep -E '<TestDuration>[0-9]+.[0-9]+</TestDuration>' /tmp/artifacts/results.xml | sed -e 's/<TestDuration>//' | sed -e 's/<\/TestDuration>//') @@ -215 +256,5 @@ In addition to a container file, the directory contains the following bash scrip - xmlstarlet ed -L -u /FinalStatus/TestDuration -v $CALCULATED_DURATION /tmp/artifacts/results.xml + sed -i.bak.td 's/<TestDuration>[0-9]*\.[0-9]*<\/TestDuration>/<TestDuration>'"$CALCULATED_DURATION"'<\/TestDuration>/' /tmp/artifacts/results.xml + fi + + if [ "$TEST_TYPE" == "simple" ]; then + TEST_TYPE="jmeter" @@ -221,3 +266,5 @@ In addition to a container file, the directory contains the following bash scrip - aws s3 cp /tmp/artifacts/jmeter.log s3://$S3_BUCKET/results/${TEST_ID}/jmeter-${PREFIX}-${UUID}-${AWS_REGION}.log --region $MAIN_STACK_REGION - aws s3 cp /tmp/artifacts/jmeter.out s3://$S3_BUCKET/results/${TEST_ID}/jmeter-${PREFIX}-${UUID}-${AWS_REGION}.out --region $MAIN_STACK_REGION - aws s3 cp /tmp/artifacts/jmeter.err s3://$S3_BUCKET/results/${TEST_ID}/jmeter-${PREFIX}-${UUID}-${AWS_REGION}.err --region $MAIN_STACK_REGION + aws s3 cp /tmp/artifacts/$LOG_FILE s3://$S3_BUCKET/results/${TEST_ID}/${TEST_TYPE}-${PREFIX}-${UUID}-${AWS_REGION}.log --region $MAIN_STACK_REGION + aws s3 cp /tmp/artifacts/$OUT_FILE s3://$S3_BUCKET/results/${TEST_ID}/${TEST_TYPE}-${PREFIX}-${UUID}-${AWS_REGION}.out --region $MAIN_STACK_REGION