AWS solutions documentation change
Summary
Updated cost estimates and breakdown for migration scenarios, shifted from ECS to EKS deployment model, added backfill-only migration option, and included cost optimization tips.
Security assessment
The changes exclusively focus on cost calculations, deployment architecture comparisons (ECS vs EKS), and optimization strategies. There is no mention of vulnerabilities, security patches, access controls, encryption, or any security-related features. The VPC endpoint references are for cost reduction, not security enhancements.
Diff
diff --git a/solutions/latest/migration-assistant-for-amazon-opensearch-service/cost.md b/solutions/latest/migration-assistant-for-amazon-opensearch-service/cost.md index 188676a90..57af08e64 100644 --- a//solutions/latest/migration-assistant-for-amazon-opensearch-service/cost.md +++ b//solutions/latest/migration-assistant-for-amazon-opensearch-service/cost.md @@ -6,0 +7,2 @@ +Cost optimization tipsDeprecated: Legacy Amazon ECS deployment cost estimate + @@ -9 +11 @@ -You are responsible for the cost of the AWS services used while running this solution. As of this revision, the cost for running this solution with the default settings in the US East (N. Virginia) Region is approximately **$3,096 for a 15-day migration with 100 TB of existing data and 15 MBps of live traffic**. These costs are for the resources shown in the sample cost table. +You are responsible for the cost of the AWS services used while running this solution. As of this revision, the cost for running this solution with the default settings in the US East (N. Virginia) Region is approximately **$2,510 for a zero-downtime migration over 15 days with 100 TB of existing data and 15 MBps of live traffic**. A **backfill-only migration** of the same 100 TB costs approximately **$1,430 over 7 days** because it eliminates the Capture Proxy, Kafka, and Traffic Replayer components entirely. In this model, the initial snapshot and bulk RFS backfill run while the source cluster remains live (no downtime). The maintenance window is limited to the time required to pause ingestion, take a final incremental snapshot of the delta (changes accumulated during backfill), migrate that delta with RFS at higher parallelism, and cut over — typically **1–2 hours** depending on the write volume to the source during the backfill period. These costs are for the resources shown in the sample cost table. @@ -15,4 +16,0 @@ A migration typically has different duration and data volume for the individual -###### Note - -The cost example below is based on the legacy ECS deployment of Migration Assistant. The Amazon EKS deployment uses different compute primitives — workloads run as short-lived pods on Amazon EKS managed node groups or Karpenter-provisioned capacity instead of AWS Fargate tasks on Amazon ECS. The data-volume cost components (Amazon S3 snapshot storage, Amazon MSK or Strimzi-on-EKS Kafka throughput, Amazon EFS for replay tuples, Amazon CloudWatch logs and metrics) and the time-based shape are similar; the per-pod compute cost varies with the EKS instance types and node-pool sizing you choose. - @@ -47,2 +44,0 @@ To understand the costs, we need to map these steps into cost components, includ - * Target Proxy - 4 days - @@ -60 +56,7 @@ Assumptions: - * The Source Cluster, Target Cluster, and VPC are imported and cost excluded from this calculation, including data transfer cost based on VPC setup. We recommend that the VPC includes an Amazon S3 Gateway Endpoint and cluster data communication over VPC Interface Endpoints to reduce data transfer cost. + * The Source Cluster, Target Cluster, and VPC are imported and cost excluded from this calculation, including data transfer cost based on VPC setup. The VPC includes an Amazon S3 Gateway Endpoint (free) and VPC Interface Endpoints for Amazon CloudWatch Logs, Amazon EFS, and Amazon ECR. + + * Amazon EKS Auto Mode pod compute pricing: $0.04048 per vCPU-hour and $0.004445 per GiB-hour (On-Demand, x86_64, Linux). + + * Apache Kafka is provided by Strimzi on Amazon EKS with Amazon EBS gp3 persistent volumes (replaces Amazon MSK). + + * All pods run on Amazon EKS Auto Mode managed nodes. @@ -69,3 +71,10 @@ AWS service | Dimensions | Cost [USD] -**Core services** | | -Amazon ECS | Migration Management Console: Task vCPU: _1 task × 24 hours per day × 15 days × (0.50 vCPU × 0.04048 USD per hour) = $7.29_ Task Memory: _1 task × 24 hours per day × 15 days × (1 GB × 0.004445 USD per hour) = $1.60_ | $8.89 -AWS Lambda | _< 10 requests × 0.20 per 1M requests = ~$0.00_ _< 10 seconds × $0.0000166667 per GB-second = ~$0.00_ | ~$0.00 +**Core services (15 days)** | | +Amazon EKS | Cluster fee: _$0.10 per hour × 24 hours per day × 15 days = $36.00_ | $36.00 +Amazon EKS (pod compute) | Migration Console (1.5 vCPU, 2 GiB): _vCPU: 1.5 × $0.04048 × 24 × 15 = $21.86_ _Memory: 2 × $0.004445 × 24 × 15 = $3.20_ | $25.06 +Amazon EKS (pod compute) | Platform pods (Argo Workflows controller and server, Strimzi operator, cert-manager, OTEL Collector, Fluent Bit, Prometheus): _Total: 3.5 vCPU, 6.4 GiB aggregate resource requests_ _vCPU: 3.5 × $0.04048 × 24 × 15 = $50.93_ _Memory: 6.4 × $0.004445 × 24 × 15 = $10.24_ [NOTE] ==== Platform pods provide workflow orchestration (Argo Workflows), Kafka operator (Strimzi), TLS certificate management (cert-manager), metrics and trace collection (OTEL Collector), log forwarding (Fluent Bit), and metrics storage (Prometheus). ==== | $37.71 +Amazon EBS | Prometheus persistent volume (50 GiB, gp3): _(50 × $0.08 per GiB-month × 15 days) / 30 days per month = $2.00_ | $2.00 +Amazon Virtual Private Cloud (Amazon VPC) | NAT Gateway (2 Availability Zones): _2 × $0.045 per hour × 24 hours per day × 15 days = $32.40_ _Data processed: 30 GB × $0.045 per GB = $1.35_ VPC Interface Endpoints (CloudWatch Logs, Amazon EFS, Amazon ECR API, Amazon ECR Docker — 2 AZs each): _4 endpoints × 2 AZs × $0.01 per hour × 24 hours per day × 15 days = $28.80_ + +###### Note + +Amazon S3 Gateway Endpoint is free. VPC Interface Endpoints reduce data transfer cost for in-VPC traffic. | $62.55 @@ -73,7 +82,2 @@ AWS Secrets Manager | _($0.40 per secret-month × 10 secrets × 15 days) / 30 d -Amazon Route 53 | _($0.50 per Hosted Zone × 15 days) / 30 days per month = $0.25_ | $0.25 -Amazon Elastic Container Registry (Amazon ECR) | _(3 GB × $0.10 GB/month × 15 days) / 30 days per month = $0.15_ | $0.15 -Amazon EC2 | Bootstrap Box (t3.large): _1 × $0.0832/hr × 24 hours per day × 15 days = $29.95_ | $29.95 -Amazon EBS | Bootstrap Box: _(50 GB × $0.08/GB-month × 15 days) / 30 days per month = $2.00_ | $2.00 -Amazon Virtual Private Cloud (Amazon VPC) | Bootstrap Box VPC - NAT Gateway: _1 × 24 hours per day × 15 days × $0.045/hour = $16.20_ _30 GB × $0.045/GB = $1.35_ | $17.55 -Elastic Load Balancing | Application Load Balancer hours: _15 days × 24 hours × $0.0225 per Application Load Balancer-hour = $8.10_ | $8.10 -**Core services total:** | | **$68.89** +Amazon Elastic Container Registry (Amazon ECR) | _(3 GiB × $0.10 per GiB-month × 15 days) / 30 days per month = $0.15_ | $0.15 +**Core services total:** | | **$165.47** @@ -92,3 +96,6 @@ AWS X-Ray | _1M traces × $5.00 per million traces = $5.00_ | $5.00 -**Optional services** | | -**Capture proxy (7.776 TB), 6 days running; 4 days data retention)** | | -Amazon Managed Streaming for Apache Kafka (Amazon MSK) | +**Capture proxy and Kafka (7.776 TB captured over 6 days; 4 days data retention)** | | +Amazon EKS (pod compute) | Capture Proxy pods (4 pods × 0.5 vCPU, 2 GiB each, 6 days): _vCPU: 4 × 0.5 × $0.04048 × 24 × 6 = $11.66_ _Memory: 4 × 2 × $0.004445 × 24 × 6 = $5.12_ | $16.78 +Elastic Load Balancing | Network Load Balancer (6 days): _NLB hours: $0.0225 per hour × 24 × 6 = $3.24_ _NLB Capacity Units (processed bytes): 7,776 GB / (6 × 24 hours) = 54 NLCUs average × $0.006 per NLCU-hour × 144 hours = $46.66_ | $49.90 +Amazon EKS (pod compute) | Strimzi Kafka broker pods (3 brokers × 1 vCPU, 4 GiB each, 10 days): _vCPU: 3 × 1 × $0.04048 × 24 × 10 = $29.15_ _Memory: 3 × 4 × $0.004445 × 24 × 10 = $12.80_ | $41.95 +Amazon EKS (pod compute) | Strimzi ZooKeeper pods (3 nodes × 0.5 vCPU, 1 GiB each, 10 days): _vCPU: 3 × 0.5 × $0.04048 × 24 × 10 = $14.57_ _Memory: 3 × 1 × $0.004445 × 24 × 10 = $3.20_ | $17.77 +Amazon EBS | Kafka persistent volumes (gp3): @@ -98,8 +105,2 @@ Amazon Managed Streaming for Apache Kafka (Amazon MSK) | -Assume deployment in 2 Availability Zones. Broker (6 days running + 4 days data retention): _ 2 nodes (M5.large) × $0.21 node per hour × 10 days × 24 hours per day = $151.20 _ Storage (9,331 GB Provisioned (20% buffer) for 6 days running + 4 days data retention): _(2 nodes × 9,331 GB × $0.10 per GB-month × 10 days) / 30 days per month = $620.07 _ Data transfer: _7,776 GB × $0.01 per GB inbound = $77.76_ | $849.03 -Amazon ECS | Task vCPU: _4 tasks × 0.5 vCPU × 0.04048 USD per vCPU-hour × 6 days × 24 hours per day = $11.66 __ Task memory:_4 tasks × 2 GB × 0.004445 USD per GB-hour × 6 days × 24 hours per day = $5.12 | $16.78 -Elastic Load Balancing | Load Balancer Capacity Unity (LCU) (assuming GB is the cost dimension): _7,776 GB × $0.008 per LCU = $62.21_ | $62.21 -**Capture proxy total:** | | **$928.02** -**Target proxy (15 MBps for 4 days (5.184 TB))** | | -Amazon ECS | Task vCPU: _4 tasks × 0.5 vCPU × 0.04048 USD per vCPU-hour) × 4 days × 24 hours per day = $7.77_ Task memory: _4 tasks × 2 GB × 0.004445 USD per GB-hour × 4 days × 24 hours per day = $3.41_ | $11.18 -Elastic Load Balancing | LCU (assuming GB is the cost dimension): _5,184 GB × $0.008 per LCU = $41.47_ | $41.47 -**Target proxy total:** | | **$52.65** +Storage is sized for peak capture volume with 20% buffer. With replication factor 1 (recommended for transient migration data), total storage is distributed across brokers. _9,331 GiB provisioned × $0.08 per GiB-month × 10 days / 30 days per month = $248.83_ | $248.83 +**Capture proxy and Kafka total:** | | **$375.23** @@ -107,2 +108 @@ Elastic Load Balancing | LCU (assuming GB is the cost dimension): _5,184 GB × -Amazon ECS | Task vCPU: _1 task × 1 vCPU × 0.04048 USD per vCPU-hour × 2 days × 24 hours per day = $1.94_ Task memory: _1 task × 4 GB × 0.004445 USD per GB-hour × 2 days × 24 hours per day = $0.86_ | $2.80 -Amazon MSK | Intra-Region: _7,776 GB × 0.01 USD per GB outbound = $77.76_ | $77.76 +Amazon EKS (pod compute) | Traffic Replayer pod (1 pod × 1 vCPU, 4 GiB, 2 days): _vCPU: 1 × $0.04048 × 24 × 2 = $1.94_ _Memory: 4 × $0.004445 × 24 × 2 = $0.85_ | $2.79 @@ -113,4 +113,4 @@ Amazon Elastic File System (Amazon EFS) | -Calculating cost based on constant size of 15.552 TB (7.776 TB × 2). Actual cost might differ due to some data spending less than 5 days in EFS Infrequent Access or data inflation. EFS Standard (infrequent after 1 day): _15,552 GB × $0.30 per GB-month × 1 day / 30 days per month = $155.52_ EFS Infrequent Access (5 days maximum): _15,552 GB × $0.025 per GB-month × 5 days / 30 days per month = $64.80_ EFS Infrequent Access - Tiering: _15,552 GB × $0.01 per GB = $155.52_ | $375.84 -**Replay total:** | | **$456.40** -**Historical Backfill (100 TB Snapshot over 1 month)** | | -Amazon ECS | RFS: Each task is capable of delivering approximately 5 MBps of snapshot data. To deliver 100 TB over 3 days, 77 tasks are needed. Task vCPU: _77 tasks × 3 days × 24 hours per day × 2 vCPUs × $0.04048 per hour = $448.84 __ Task memory:_77 tasks × 3 days × 24 hours per day × 4 GB × $0.004445 per hour = $98.57 Task storage: _77 tasks × 3 days × 24 hours per day × (200GB - 20GB Free Tier) × $0.000111 per hour = $110.77 _ | $657.34 +Replay tuples store source and target request/response pairs. Volume = 7.776 TB × 2 = 15.552 TB. Actual cost might differ due to some data spending less than 5 days in EFS Infrequent Access or data inflation. EFS Standard (infrequent after 1 day): _15,552 GiB × $0.30 per GiB-month × 1 day / 30 days per month = $155.52_ EFS Infrequent Access (5 days maximum): _15,552 GiB × $0.025 per GiB-month × 5 days / 30 days per month = $64.80_ EFS Infrequent Access - Tiering: _15,552 GiB × $0.01 per GiB = $155.52_ | $375.84 +**Replay total:** | | **$378.63** +**Historical Backfill (100 TB snapshot, 3 days RFS)** | | +Amazon EKS (pod compute) | RFS worker pods: Each pod delivers approximately 5 MBps of snapshot data. To deliver 100 TB over 3 days, 77 pods are needed. _vCPU: 77 pods × 2 vCPU × $0.04048 × 24 × 3 = $448.84_ _Memory: 77 pods × 4 GiB × $0.004445 × 24 × 3 = $98.57_ _Ephemeral storage: 77 pods × (200 GiB − 20 GiB free tier) × $0.000111 per GiB-hour × 24 × 3 = $110.77_ | $658.18 @@ -121,3 +121,3 @@ Amazon S3 | -This calculates the cost for all the data occurring in the S3 bucket for the entire duration. The actual cost might be less for storage due to partial rate while the snapshot is being taken.Snapshot storage: _(100 TB × 1,000 GB per TB × $0.023 per GB-month × 10 days) / 30 days per month = $766.66_ | $766.66 -**Historical Backfill total:** | | **$1,424.00** -**Total for all components:** | | **$3,095.41** +This calculates the cost for all the data residing in the S3 bucket for the entire duration. The actual cost might be less for storage due to partial rate while the snapshot is being taken. Snapshot storage: _(100 TB × 1,000 GiB per TB × $0.023 per GiB-month × 10 days) / 30 days per month = $766.67_ | $766.67 +**Historical Backfill total:** | | **$1,424.85** +**Total for all components:** | | **$2,509.63** @@ -125 +125 @@ This calculates the cost for all the data occurring in the S3 bucket for the ent -The Capture and Replay yields an effective cost per TB of $199.87, and the Historic Backfill is $15.41 per TB. For a given cluster, the amount of data in Historic Backfill is largely fixed, while the data for Capture and Replay is based on the time needed to capture. For this reason, it can be cost advantageous to scale up the target OpenSearch cluster beyond the final intended capacity for the Historic Backfill period to reduce the duration and quantity of data for Capture and Replay. +The Capture and Replay yields an effective cost per TB of $97.14, and the Historic Backfill is $14.25 per TB. For a given cluster, the amount of data in Historic Backfill is largely fixed, while the data for Capture and Replay is based on the time needed to capture. For this reason, it can be cost advantageous to scale up the Amazon OpenSearch Service domain beyond the final intended capacity for the Historic Backfill period to reduce the duration and quantity of data for Capture and Replay. @@ -127 +127 @@ The Capture and Replay yields an effective cost per TB of $199.87, and the Histo -Taking the table above and dividing out the components cost, we can determine the following calculation method for cost. This is a broad estimation making assumptions on traffic pattern including size per request, and number of new connections. We can also subdivide this by migration type applicable. +Taking the table above and dividing out the component costs, we can determine the following calculation method for cost. This is a broad estimation making assumptions on traffic pattern including size per request and number of new connections. We can also subdivide this by migration type applicable. @@ -138 +138 @@ Core services | -| $4.59/day +| $11.03/day @@ -152 +152 @@ Capture runtime | -| $17.22/day + $8.00/TB-day + $18.00/TB +| $9.31/day + $3.20/TB-day (Kafka EBS) + $6.00/TB (NLB) @@ -158 +158 @@ Capture data retention period | -| $15.12/day + $8.00/TB-day +| $5.97/day + $3.20/TB-day (Kafka EBS) @@ -176 +176 @@ Reindex from snapshot | -| $6.57/TB +| $6.58/TB @@ -182 +182 @@ Traffic Replayer | -| $1.40/day + $50.00/TB + $1.6667/TB-day +| $1.40/day + $40.00/TB (EFS) + $1.6667/TB-day (EFS IA) @@ -186 +186 @@ Traffic Replayer | -Due to Amazon EFS-Intelligent Tiering, $1.6667/TB-day is not paid for the first day. +Due to Amazon EFS Intelligent-Tiering, $1.6667/TB-day is not paid for the first day. @@ -189 +189 @@ Traffic Replayer data retention period | - * Historical only + * Live only @@ -193 +192,0 @@ Traffic Replayer data retention period | -Target proxy | @@ -195,2 +194 @@ Target proxy | - * Historical only - * Full migration +We can also identify the cost for full migrations of different sizes that follow the 15-day migration timeline. By applying the durations shown previously, as well as a conversion from TB/day to MBps, we get the following formula for a representative cost estimate: @@ -198,5 +196 @@ Target proxy | -| $2.10/day + $8.00/TB - -We can also identify the cost for full migrations of different sizes that follow the 15-day migration timeline. By applying the durations shown previously, as well as a conversion from TB/day to MB/s, we get the following formula for a representative cost estimate: - -_15-day migration = $390.35 + $14.24 × HistoricalTB + $71.86 × LiveMBps_ +_15-day migration ≈ $413 + $14.25 × HistoricalTB + $44.76 × LiveMBps_ @@ -209,0 +204,30 @@ All costs are rounded to the nearest dollar. +15-day full migration cost ($ USD/migration) | | Live traffic throughput | +---|---|---|--- +Historical data volume | 5 MBps | 20 MBps | 50 MBps +1 TB | $651 | $1,322 | $2,665 +10 TB | $780 | $1,451 | $2,794 +100 TB | $2,062 | $2,733 | $4,076 + +## Cost optimization tips + + * **Spot Instances for RFS workers** : RFS workers are stateless and checkpoint progress. Configure the Karpenter NodePool with `capacity-spread` to use Spot Instances for up to 70% savings on backfill compute. + + * **Scale the target cluster during backfill** : A larger target processes documents faster, reducing the capture window and overall Kafka storage and EFS costs. + + * **Right-size Kafka retention** : If replay catches up quickly, reduce the Kafka log retention to free EBS storage sooner. + + * **Use VPC Interface Endpoints** : Already included in this estimate — these reduce NAT Gateway data processing charges for traffic to AWS services. + + + + +## Deprecated: Legacy Amazon ECS deployment cost estimate + +###### Note + +The following cost estimate applies to the legacy Amazon ECS deployment of Migration Assistant (version 2.x and earlier). This deployment model uses AWS Fargate tasks on Amazon ECS, Amazon MSK for Kafka, and Application Load Balancers. It is provided for reference only. New deployments should use the Amazon EKS deployment described above. + +The legacy ECS deployment of the same 15-day, 100 TB, 15 MBps scenario costs approximately **$3,096** with the following formula: + +_15-day migration = $390.35 + $14.24 × HistoricalTB + $71.86 × LiveMBps_ + @@ -216,0 +241,2 @@ Historical data volume | 5 MBps | 20 MBps | 50 MBps +The primary cost difference between the EKS and legacy ECS deployments comes from replacing Amazon MSK with Strimzi Kafka on Amazon EKS (EBS gp3 storage at $0.08/GiB-month versus MSK storage at $0.10/GiB-month with per-broker duplication) and eliminating intra-region MSK data transfer charges. The EKS deployment adds a fixed cluster fee ($0.10/hr) and VPC Interface Endpoint charges, but these are offset by the Kafka savings at moderate-to-high traffic volumes. +