AWS solutions documentation change
Summary
Updated deployment documentation to focus on EKS-based deployment, added prerequisites, detailed bootstrap script usage, step-by-step deployment instructions, authentication methods, and legacy ECS deprecation notice.
Security assessment
Added documentation for authentication methods including Kubernetes secrets for basic auth and IRSA for SigV4 authentication, which are security features. No evidence of addressing a specific vulnerability.
Diff
diff --git a/solutions/latest/migration-assistant-for-amazon-opensearch-service/deploy-the-solution.md b/solutions/latest/migration-assistant-for-amazon-opensearch-service/deploy-the-solution.md index e62906a8c..33ce8c2d6 100644 --- a//solutions/latest/migration-assistant-for-amazon-opensearch-service/deploy-the-solution.md +++ b//solutions/latest/migration-assistant-for-amazon-opensearch-service/deploy-the-solution.md @@ -7 +7 @@ -Deployment options +PrerequisitesDeployment optionsStep 1: Download the bootstrap scriptStep 2: Deploy into a new or existing VPCStep 3: Verify the deploymentStep 4: Access the Migration ConsoleStep 5: Use the AWS helpers the deployment created for youAuthentication on Amazon EKSPrivate or isolated networksGrant kubectl access to a CI role or teammateRecovery if the bootstrap failsLegacy Amazon ECS deployment (support only) @@ -11 +11 @@ Deployment options -This solution provides two deployment options: ECS-based deployment and EKS-based deployment. Choose the option that best fits your migration requirements. +This solution deploys to Amazon Elastic Kubernetes Service (Amazon EKS). The Amazon EKS deployment is the recommended deployment for Migration Assistant 3.0 and is the deployment used throughout the rest of this implementation guide. A legacy Amazon ECS deployment remains available in support-only mode for existing deployments. @@ -21 +21,16 @@ To update to a newer version of this solution, [uninstall](./uninstall-the-solut -This solution sends operational metrics to AWS. We use this data to better understand how customers use this solution and related services and products. AWS owns the data gathered though this survey. Data collection is subject to the [Privacy Notice](https://aws.amazon.com/privacy/). +This solution sends operational metrics to AWS. We use this data to better understand how customers use this solution and related services and products. AWS owns the data gathered through this survey. Data collection is subject to the [Privacy Notice](https://aws.amazon.com/privacy/). + +## Prerequisites + +To deploy this solution on Amazon EKS, you need: + + * An AWS account with permissions for AWS CloudFormation, Amazon EKS, AWS IAM, Amazon EC2, Amazon ECR, Amazon S3, Amazon CloudWatch, AWS Secrets Manager, and related services. + + * Either [AWS CloudShell](https://aws.amazon.com/cloudshell/) or a local terminal with AWS CLI v2, `kubectl`, and Helm 3 installed. AWS CloudShell is recommended because it comes pre-configured with the required tools and avoids platform-specific issues (for example, the `tac` command used by the bootstrap script is not available on macOS by default). + + * Network connectivity from the Amazon EKS cluster to your source cluster and your Amazon OpenSearch Service domain or Amazon OpenSearch Serverless collection. + + + + +The bootstrap script handles cluster creation, IAM role wiring (IRSA), private image mirroring, VPC endpoint setup, and Helm chart installation. @@ -25 +40 @@ This solution sends operational metrics to AWS. We use this data to better under -### EKS deployment (recommended) +Choose one of the following CloudFormation templates to deploy the Amazon EKS infrastructure: @@ -27 +42 @@ This solution sends operational metrics to AWS. We use this data to better under -The EKS-based deployment is the recommended deployment option for Migration Assistant 3.0. It leverages Amazon Elastic Kubernetes Service (Amazon EKS) to provide a simplified workflow experience with faster configuration changes and built-in orchestration. + * [Launch with a new VPC](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service-create-vpc.template=ImplementationGuide) — Creates a new VPC with the required networking, public and private subnets, NAT gateway, and Amazon EKS cluster. @@ -29 +44 @@ The EKS-based deployment is the recommended deployment option for Migration Assi -#### Launch the CloudFormation stack + * [Launch into an existing VPC](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service-import-vpc.template=ImplementationGuide) — Uses an existing VPC. The selected subnets must be in different Availability Zones and must allow outbound connectivity (either through a NAT gateway or VPC endpoints, depending on whether you are deploying into isolated subnets). @@ -31 +45,0 @@ The EKS-based deployment is the recommended deployment option for Migration Assi -Choose one of the following options to deploy the EKS infrastructure: @@ -33 +46,0 @@ Choose one of the following options to deploy the EKS infrastructure: - * [Launch with a new VPC](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service-create-vpc-eks.template) — Creates a new VPC with all required networking. @@ -35 +47,0 @@ Choose one of the following options to deploy the EKS infrastructure: - * [Launch into an existing VPC](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://solutions-reference.s3.amazonaws.com/migration-assistant-for-amazon-opensearch-service/latest/migration-assistant-for-amazon-opensearch-service-import-vpc-eks.template&redirectId) — Uses an existing VPC (requires proper subnet and security group configuration). @@ -36,0 +49 @@ Choose one of the following options to deploy the EKS infrastructure: +After the CloudFormation stack deploys, run the bootstrap script to install the Migration Assistant Helm chart and configure the Amazon EKS cluster. @@ -37,0 +51 @@ Choose one of the following options to deploy the EKS infrastructure: +## Step 1: Download the bootstrap script @@ -38,0 +53 @@ Choose one of the following options to deploy the EKS infrastructure: +Throughout this guide, `STAGE` is a short label such as `dev`, `staging`, or `prod`. It is used in cluster and resource names so you can keep multiple deployments separate. @@ -40 +54,0 @@ Choose one of the following options to deploy the EKS infrastructure: -After the CloudFormation stack deploys, run the bootstrap script to install the Helm chart and configure the EKS cluster. For detailed instructions, see [Deploying to EKS on GitHub](https://github.com/opensearch-project/opensearch-migrations/wiki/Deploying-to-EKS). @@ -42 +56,3 @@ After the CloudFormation stack deploys, run the bootstrap script to install the -The deployment guide provides detailed instructions for: + curl -sL -o aws-bootstrap.sh \ + "https://github.com/opensearch-project/opensearch-migrations/releases/latest/download/aws-bootstrap.sh" \ + chmod +x aws-bootstrap.sh @@ -44 +60 @@ The deployment guide provides detailed instructions for: - * Prerequisites and environment setup +Run `./aws-bootstrap.sh --help` to see every available flag for the version you downloaded. @@ -46 +62 @@ The deployment guide provides detailed instructions for: - * CloudFormation-based infrastructure deployment (new VPC or existing VPC) +### Bootstrap flag reference @@ -48 +64 @@ The deployment guide provides detailed instructions for: - * Helm chart installation and configuration +The flags below cover the most common cases. The script always installs the matching Helm chart and Amazon CloudWatch dashboards after the AWS CloudFormation stack succeeds. @@ -50 +66,26 @@ The deployment guide provides detailed instructions for: - * Migration execution using the Workflow CLI +Group | Flag | Purpose +---|---|--- +**Mode (one required)** | `--deploy-create-vpc-cfn` | Create a new VPC and Amazon EKS cluster +| `--deploy-import-vpc-cfn` | Reuse an existing VPC. Requires `--vpc-id` and `--subnet-ids` +| `--skip-cfn-deploy` | Skip AWS CloudFormation. Use when the stack already exists and you only want to re-bootstrap the cluster +**Identity** | `--stack-name name` | AWS CloudFormation stack name (required with `--deploy-*-cfn`) +| `--stage name` | Short label for cluster and resource names. Defaults to `dev` +| `--region region` | AWS Region +| `--vpc-id id` | Existing VPC ID (with `--deploy-import-vpc-cfn`) +| `--subnet-ids id1,id2` | Comma-separated subnets in different Availability Zones (with `--deploy-import-vpc-cfn`) +**Versioning** | `--version tag` | Pin to a published GitHub release tag (find tags at [OpenSearch Migrations Releases](https://github.com/opensearch-project/opensearch-migrations/releases)). Use this for reproducible deployments. +| `--build` | Build all artifacts from source (requires a repository checkout). Mutually exclusive with `--version` +**Networking** | `--create-vpc-endpoints` | Create the seven VPC endpoints needed for isolated subnets (Amazon S3, Amazon ECR API, Amazon ECR Docker, Amazon CloudWatch Logs, Amazon EFS, AWS STS, Amazon EKS Auth) +| `--use-public-images` | Skip mirroring images into private Amazon ECR. Use only when the cluster has internet access and you do not want a private mirror +| `--ma-images-source registry` | Copy Migration Assistant images from another Amazon ECR registry. Useful when images were built on a separate cluster with internet access +**Access** | `--eks-access-principal-arn arn` | Grant a CI role or teammate cluster-admin access. Combine with `--skip-cfn-deploy --skip-console-exec` to grant access without redeploying +| `--kubectl-context name` | Set a custom alias for the kubectl context (defaults to the Amazon EKS cluster name) +| `--skip-setting-k8s-context` | Don’t switch your active kubectl context to the new cluster +| `--skip-console-exec` | Don’t auto-exec into the Migration Console pod when the script finishes +**TLS (Capture proxy)** | `--tls-mode none` | No TLS termination on the capture proxy (default) +| `--tls-mode self-signed` | cert-manager-issued self-signed certificate +| `--tls-mode pca-existing --pca-arn arn` | Use an existing AWS Private CA +| `--tls-mode pca-create` | Create a new AWS Private CA via ACK +**Helm** | `--namespace name` | Override the Migration Assistant namespace (default: `ma`) +| `--helm-values path` | Extra values file for the Helm install — for example, to customize `workloadsNodePool.architectures` +| `--use-general-node-pool` | Use the Amazon EKS Auto Mode general-purpose pool instead of the production Karpenter NodePool @@ -52 +93 @@ The deployment guide provides detailed instructions for: - * Monitoring and troubleshooting +## Step 2: Deploy into a new or existing VPC @@ -53,0 +95 @@ The deployment guide provides detailed instructions for: +### New VPC (latest published release) @@ -55,0 +98,5 @@ The deployment guide provides detailed instructions for: + ./aws-bootstrap.sh \ + --deploy-create-vpc-cfn \ + --stack-name MA \ + --stage dev \ + --region us-east-2 @@ -57 +104 @@ The deployment guide provides detailed instructions for: -For the complete Migration Assistant documentation, see [Migration Assistant for OpenSearch](https://docs.opensearch.org/latest/migration-assistant/). +### New VPC pinned to a specific release @@ -59 +105,0 @@ For the complete Migration Assistant documentation, see [Migration Assistant for -### ECS deployment (support only) @@ -61 +107,67 @@ For the complete Migration Assistant documentation, see [Migration Assistant for -###### Important + ./aws-bootstrap.sh \ + --deploy-create-vpc-cfn \ + --stack-name MA \ + --stage prod \ + --region us-east-2 \ + --version 3.0.1 + +###### Note + +Pinning a version makes the deployment reproducible. Customers and CI runs that need to deploy the same artifacts again should always pass `--version`. + +### Existing VPC + + + ./aws-bootstrap.sh \ + --deploy-import-vpc-cfn \ + --stack-name MA \ + --stage dev \ + --vpc-id vpc-0abc123 \ + --subnet-ids subnet-111,subnet-222 \ + --region us-east-2 + +When the script finishes, it has already installed the Helm chart and configured the core platform pieces. + +## Step 3: Verify the deployment + +Update your kubeconfig to point at the new Amazon EKS cluster: + + + aws eks update-kubeconfig --region REGION --name migration-eks-cluster-STAGE-REGION + +Then list the pods in the `ma` namespace: + + + kubectl get pods -n ma + +You should see the Migration Console (`migration-console-0`), the Argo workflow controller, and the Argo server in `Running` state. + +## Step 4: Access the Migration Console + + + kubectl exec -it migration-console-0 -n ma -- /bin/bash + +Once you are in the Migration Console pod, the migration flow is the same regardless of source: confirm the version, load the sample workflow configuration, run a pilot, validate, and then run the full migration. See [Use the solution](./use-the-solution.html). + +## Step 5: Use the AWS helpers the deployment created for you + +The Amazon EKS deployment provides a default Amazon S3 bucket and snapshot-role wiring so you do not have to build this manually. + +### Default Amazon S3 bucket + +The deployment creates a default Amazon S3 bucket for migration artifacts and snapshots: + + + s3://migrations-default-ACCOUNT_ID-STAGE-REGION + +The default bucket is mounted on the Migration Console pod for direct artifact access. + +### Snapshot role output + +If your workflow needs a snapshot role ARN, look it up from the AWS CloudFormation outputs: + + + aws cloudformation describe-stacks \ + --stack-name YOUR_STACK_NAME \ + --query "Stacks[0].Outputs[?contains(OutputKey,'MigrationsExportString')].OutputValue" \ + --output text @@ -63 +175,83 @@ For the complete Migration Assistant documentation, see [Migration Assistant for -The ECS-based deployment is in support-only mode and will reach end of support on **July 15, 2026**. No new features will be developed for the ECS deployment. New deployments should use the EKS deployment option. +## Authentication on Amazon EKS + +### Basic authentication + +For sources that require basic authentication, create a Kubernetes secret in the `ma` namespace and reference it in `authConfig.basic.secretName` in your workflow configuration: + + + kubectl create secret generic source-credentials \ + --from-literal=username=SOURCE_USER \ + --from-literal=password=SOURCE_PASSWORD \ + -n ma + +### AWS Signature Version 4 (SigV4) + +For Amazon OpenSearch Service domains, Amazon OpenSearch Serverless collections, and other AWS services that authenticate with SigV4, the Amazon EKS stack uses [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to give two sets of pods an AWS identity: + + * The Migration Console pod (`migration-console-0`), which runs under the `migration-console-access-role` Kubernetes service account. + + * The Argo workflow executor pods, which run under the `argo-workflow-executor` Kubernetes service account. +