AWS res high security documentation change
Summary
Added documentation for multiple known issues including: Regex failure in Cognito user registration, certificate renewal failure for custom domains, AD users unable to SSH to Bastion Host, VDI auto stop broken in isolated VPCs, failure to launch VDI for graphic instance types, AMI preparation failure, group sync failure during AD sync, and a security vulnerability (CVE-2024-6387) in OpenSSH for RHEL9 and Ubuntu VDIs.
Security assessment
The change explicitly documents CVE-2024-6387 (RegreSSHion), a critical OpenSSH vulnerability allowing remote code execution. It provides detailed mitigation steps including patching existing VDIs and updating future deployments. This constitutes concrete evidence of addressing a security vulnerability.
Diff
diff --git a/res/archive/release-minus-3/ug/res-troubleshooting-known-issues.md b/res/archive/release-minus-3/ug/res-troubleshooting-known-issues.md index db619edaa..a1be07413 100644 --- a//res/archive/release-minus-3/ug/res-troubleshooting-known-issues.md +++ b//res/archive/release-minus-3/ug/res-troubleshooting-known-issues.md @@ -10,0 +11,12 @@ Known Issues 2024.x + * (2024.12 and 2024.12.01) Regex failure when registering a new Cognito user + + * (2024.12.01 and earlier) Invalid bad cert error when connecting to VDI using a custom domain + + * (2024.12 and 2024.12.01) Active Directory users cannot SSH to Bastion Host + + * (2024.10) VDI auto stop broken for RES environments deployed in isolated VPCs + + * (2024.10 and earlier) Failure to launch VDI for Graphic enhanced instance types + + * (2024.08) Preparing Infrastructure AMI Failure + @@ -14,0 +27,4 @@ Known Issues 2024.x + * (2024.06 and earlier) Group members not synced to RES during AD sync + + * (2024.06 and earlier) CVE-2024-6387, RegreSSHion, Security Vulnerability in RHEL9 and Ubuntu VDIs + @@ -27 +42,0 @@ Known Issues 2024.x - * (2024.06 and earlier) Group members not synced to RES during AD sync @@ -29 +43,0 @@ Known Issues 2024.x - * (2024.06 and earlier) CVE-2024-6387, RegreSSHion, Security Vulnerability in RHEL9 and Ubuntu VDIs @@ -31,0 +46,239 @@ Known Issues 2024.x +## Known Issues 2024.x + +........................ + +### (2024.12 and 2024.12.01) Regex failure when registering a new Cognito user + +**Bug description** + +If you attempt to register AWS Cognito users through the web portal who have email prefixes that contain ".", such as `<firstname>.<lastname>@<company>.com`, this will result in an error stating that the Cognito username does not match the defined regex pattern. + + + +This error is caused by RES auto-generating usernames from the user's email prefix. However, usernames with "." are not valid users for VDIs in certain Linux distributions supported by RES. This fix removes any "." in the email prefix when generating a username so that the username will be valid on RES Linux VDIs. + +**Affected versions** + +RES versions 2024.12 and 2024.12.01 + +**Mitigation** + + 1. Run the following commands to download `patch.py` and `cognito_sign_up_email_fix.patch` for version 2024.12 or `cognito_sign_up_email_fix.patch` for version 2024.12.01, replacing `<output-directory>` with the directory where you want to download the patch script and patch file, and `<environment-name>` with the name of your RES environment: + + 1. The patch applies to RES 2024.12 and 2024.12.01. + + 2. The patch script requires [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), Python 3.9.16 or above, and [Boto3](https://aws.amazon.com/sdk-for-python/). + + 3. Configure the AWS CLI for the account and region where RES is deployed, and make sure that you have S3 permissions to write to the bucket created by RES. + + OUTPUT_DIRECTORY=<output-directory> + ENVIRONMENT_NAME=<environment-name> + RES_VERSION=<res-version> # either 2024.12 or 2024.12.01 + + mkdir -p ${OUTPUT_DIRECTORY} + curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patch.py --output ${OUTPUT_DIRECTORY}/patch.py + curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/cognito_sign_up_email_fix.patch --output ${OUTPUT_DIRECTORY}/cognito_sign_up_email_fix.patch + + 2. Navigate to the directory where the patch script and patch file were downloaded. Run the following patch command: + + python3 ${OUTPUT_DIRECTORY}/patch.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module cluster-manager --patch ${OUTPUT_DIRECTORY}/cognito_sign_up_email_fix.patch + + + 3. Restart the Cluster Manager instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console. + + INSTANCE_ID=$(aws ec2 describe-instances \ + --filters \ + Name=tag:Name,Values=${ENVIRONMENT_NAME}-cluster-manager \ + Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME}\ + --query "Reservations[0].Instances[0].InstanceId" \ + --output text) + + aws ec2 terminate-instances --instance-ids ${INSTANCE_ID} + + 4. Verify the Cluster Manager instance status by checking the activity of the auto scaling group starting with the name `<RES-EnvironmentName>-cluster-manager-asg`. Wait until the new instance is launched successfully. + + + + +........................ + +### (2024.12.01 and earlier) Invalid bad cert error when connecting to VDI using a custom domain + +**Bug description** + +When you deploy the [External Resources recipe](./create-external-resources.html) and RES with a custom portal domain name, CertificateRenewalNode fails to refresh the TLS certificate for VDI connection with the following error in `/var/log/user-data.log`: + + + { + "type": "urn:ietf:params:acme:error:unauthorized", + "detail": "Error finalizing order :: OCSP must-staple extension is no longer available: see https://letsencrypt.org/2024/12/05/ending-ocsp", + "status": 403 + } + +As a result, you will encounter an error that states `net::ERR_CERT_DATE_INVALID` (Chrome) or `Error code: SSL_ERROR_BAD_CERT_DOMAIN` (FireFox) when you connect to your VDIs in the RES web portal. + +**Affected versions** + +2024.12.01 and earlier + +**Mitigation** + + 1. Navigate to the EC2 console. If there is an instance named `CertificateRenewalNode-`, terminate the instance. + + 2. Navigate to the Lambda console. Open the source code of the Lambda function named `-CertificateRenewalLambda-`. Identify the line staring with `./acme.sh --issue --dns dns_aws --ocsp-must-staple --keylength 4096` and remove the `--ocsp-must-staple` argument. + + 3. Select **Deploy** and wait for the code change to take effect. + + 4. To manually trigger the Lambda function: go to the **Test** tab and then select **Test**. No additional input is required. This should create a certificate EC2 instance that updates the Certificate and PrivateKey secrets in Secret Manager. The instance will be terminated automatically once the secrets are updated. + + 5. Terminate the existing dcv-gateway instance: `<env-name>-vdc-gateway` and wait for the auto scaling group to automatically deploy a new one. + + + + +**Error details** + +Let's Encrypt is ending OCSP Support in 2025. Starting from January 30, 2025, OCSP Must-Staple requests will fail unless the requesting account has previously issued a certificate that contains the OCSP Must Staple extension. Check [ https://letsencrypt.org/2024/12/05/ending-ocsp/](https://letsencrypt.org/2024/12/05/ending-ocsp/) for more details. + +........................ + +### (2024.12 and 2024.12.01) Active Directory users cannot SSH to Bastion Host + +**Bug description** + +Active Directory users receive a permission denied error when they connect to the Bastion Host following the instructions from the RES web portal. + +The Python application that runs on the Bastion Host fails to launch the SSSD service due to a missing environment variable. As a result, AD users are unknown to the operating system and cannot log in. + +**Affected versions** + +2024.12 and 2024.12.01 + +**Mitigation** + + 1. Connect to the Bastion Host instance from the EC2 console. + + 2. Edit `/etc/environment` and add `environment_name=<res-environment-name>` as a new line under IDEA_CLUSTER_NAME. + + 3. Run the following commands on the instance: + + source /etc/environment + sudo service supervisord restart + sudo systemctl restart supervisord + + 4. Try to connect to the Bastion Host again following the instructions from the RES web portal. + + + + +........................ + +### (2024.10) VDI auto stop broken for RES environments deployed in isolated VPCs + +**Bug description** + +With the 2024.10 RES release, VDI auto stop was added for VDIs that are at idle for a certain period of time. This setting can be configured in Desktop Settings → Server → Session. + +VDI auto stop is currently not supported for RES environments deployed in isolated VPCs. + +**Affected versions** + +2024.10 + +**Mitigation** + +We are currently working on a fix that will be included in a future release. However, it is still possible to manually stop VDIs in RES environments deployed in isolated VPCs. + +........................ + +### (2024.10 and earlier) Failure to launch VDI for Graphic enhanced instance types + +**Bug description** + +When an Amazon Linux 2 - x86_64, RHEL 8 - x86_64, or RHEL 9 x86_64 VDI is launched on a graphic enhanced instance type (g4, g5), the instance will get stuck in the provisioning state. This means the instance will never get to the "Ready" state and be available for connection. + +This happens because the X Server does not properly instantiate on the instances. After you apply this patch we also suggest you increase the root volume size of your software stacks for graphics instances to 50gb to ensure there is sufficient space for installing all dependencies. + +**Affected versions** + +All RES versions 2024.10 or earlier. + +**Mitigation** + + 1. Download [ patch.py](https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/2024.10/patch_scripts/patch.py) and [ graphic_enhanced_instance_types_fix.patch](https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/2024.10/patch_scripts/patches/graphic_enhanced_instance_types_fix.patch) by replacing `<output-directory>` with the directory where you want to download the patch script and patch file and `<environment-name>` with the name of your RES environment in the command below: + + 1. The patch only applies to RES 2024.10. + + 2. The patch script requires AWS CLI v2, Python 3.9.16 or above, and Boto3. + + 3. Configure the AWS CLI for the account and region where RES is deployed, and make sure that you have S3 permissions to write to the bucket created by RES. + + OUTPUT_DIRECTORY=<output-directory> + ENVIRONMENT_NAME=<environment-name> +