AWS omics documentation change
Summary
Restructured and expanded run setup documentation with detailed step-by-step instructions, added Nextflow engine settings configuration, reorganized parameters, and improved API examples.
Security assessment
The changes emphasize security practices like IAM service roles for resource access, output bucket ownership verification, and data isolation through ephemeral storage. However, there's no evidence of addressing a specific vulnerability or incident.
Diff
diff --git a/omics/latest/dev/starting-a-run.md b/omics/latest/dev/starting-a-run.md index 64a11514e..8b96c1334 100644 --- a//omics/latest/dev/starting-a-run.md +++ b//omics/latest/dev/starting-a-run.md @@ -7 +7 @@ -HealthOmics run parametersStarting a run using the consoleStarting a run using the APIGet information about a runSpecify engine settingsVPC networking +Starting a run using the consoleStarting a run using the APISpecify Nextflow engine settingsVPC networking @@ -11 +11 @@ HealthOmics run parametersStarting a run using the consoleStarting a run using t -When you start a run, you specify the resources that HealthOmics allocates for use during the run. +When you start a run, you specify the resources that HealthOmics allocates for the run. The following settings are available: @@ -13 +13 @@ When you start a run, you specify the resources that HealthOmics allocates for u -Specify the run storage type and storage amount (for static storage). To ensure data isolation and security, HealthOmics provisions the storage at the start of each run, and deprovisions it at the end of the run. For additional information, see [Run storage types in HealthOmics workflows](./workflows-run-types.html). + 1. **Output location** – Specify an Amazon S3 URI where the output files from the run are stored. If you run a high volume of workflows concurrently, use separate Amazon S3 output URIs for each workflow to avoid bucket throttling. For more information, see [Organizing objects using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the _Amazon S3 User Guide_ and [Scale Storage Connections Horizontally](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/scale-storage-connections-horizontally.html) in the _Optimizing Amazon S3 Performance_ whitepaper. @@ -15 +15 @@ Specify the run storage type and storage amount (for static storage). To ensure -Specify an Amazon S3 location for the output files. If you run a high volume of workflows concurrently, use separate Amazon S3 output URIs for each workflow to avoid bucket throttling. For more information, see [ Organizing objects using prefixes ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the _Amazon S3 User Guide_ and [ Scale Storage Connections Horizontally ](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/scale-storage-connections-horizontally.html) in the _Optimizing Amazon S3 Performance_ whitepaper. + 2. **Service role** – Specify an IAM service role that grants HealthOmics permissions to access the resources needed for the run. Optionally, the console can create the service role for you. For more information, see [Service roles for AWS HealthOmics](./permissions-service.html). @@ -17 +17 @@ Specify an Amazon S3 location for the output files. If you run a high volume of -You can also specify the run priority. How priority impacts the run depends on whether the run is associated with a run group. For additional information, see [Run priority](./creating-run-groups.html#run-priority). + 3. **Run storage** (optional, defaults to Dynamic) – Specify the run storage type and storage amount (for static storage). To ensure data isolation and security, HealthOmics provisions the storage at the start of each run, and deprovisions it at the end of the run. For more information, see [Run storage types in HealthOmics workflows](./workflows-run-types.html). @@ -19 +19 @@ You can also specify the run priority. How priority impacts the run depends on w -If a workflow has one or more versions, you can specify a version when you start the run. If you don’t specify a version, HealthOmics starts the [default workflow version](./workflows-default-version.html). + 4. **Run priority** (optional) – Assign a priority to the run. How priority impacts the run depends on whether the run is associated with a run group. For more information, see [Run priority](./creating-run-groups.html#run-priority). @@ -21 +21,7 @@ If a workflow has one or more versions, you can specify a version when you start -When using the HealthOmics API, you can provide a unique request ID for each run. The request ID is an idempotency token that HealthOmics uses to identify duplicate requests. and starts the run only once. + 5. **Workflow version** (optional) – Select a specific workflow version for the run. If you don't specify a version, HealthOmics starts the [default workflow version](./workflows-default-version.html). + + 6. **Nextflow engine settings** (optional, Nextflow only) – Specify engine settings such as version and syntax parser for Nextflow workflows during runtime. For more information, see Specify Nextflow engine settings. + + 7. **Input parameters** (optional) – Provide the workflow input parameters as a JSON file or inline values. Required parameters are defined by the workflow's parameter template. For more information, see [HealthOmics run inputs](./workflows-run-inputs.html). + + 8. **Request ID** (optional, API and CLI only) – Provide a unique request ID for each run. The request ID is an idempotency token that HealthOmics uses to identify duplicate requests and start the run only once. @@ -23 +28,0 @@ When using the HealthOmics API, you can provide a unique request ID for each run -###### Note @@ -25 +29,0 @@ When using the HealthOmics API, you can provide a unique request ID for each run -You specify an IAM service role when you start a run. Optionally, the console can create the service role for you. For more information, see [Service roles for AWS HealthOmics](./permissions-service.html). @@ -27 +30,0 @@ You specify an IAM service role when you start a run. Optionally, the console ca -###### Topics @@ -29 +32 @@ You specify an IAM service role when you start a run. Optionally, the console ca - * HealthOmics run parameters +###### Topics @@ -35,3 +38 @@ You specify an IAM service role when you start a run. Optionally, the console ca - * Get information about a run - - * Specify engine settings + * Specify Nextflow engine settings @@ -44 +45 @@ You specify an IAM service role when you start a run. Optionally, the console ca -## HealthOmics run parameters +## Starting a run using the console @@ -46 +47 @@ You specify an IAM service role when you start a run. Optionally, the console ca -When you start a run, you specify run inputs in the run parameters JSON file or you can enter the parameter values inline. For information about managing the size of the run parameters JSON file, see [Managing run parameters size](./workflows-run-inputs.html#run-input-file-options). +The Start run wizard has four steps: @@ -48 +49 @@ When you start a run, you specify run inputs in the run parameters JSON file or -HealthOmics supports the following JSON types for parameter values. + 1. Specify run details @@ -50,8 +51,5 @@ HealthOmics supports the following JSON types for parameter values. -JSON type | Example key and value | Notes ----|---|--- -boolean | "b":true | Value is not in quotes, and all lowercase. -integer | "i":7 | Value is not in quotes. -number | "f":42.3 | Value is not in quotes. -string | "s":"characters" | Value is in quotes. Use string type for text values and URIs. The URI target must be the expected input type. -array | "a":[1,2,3] | Value is not in quotes. Array members must each have the type defined by the input parameter. -object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or Struct + 2. Add parameter values + + 3. Add run group, run cache, and tags + + 4. Review and start run @@ -59 +56,0 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or -## Starting a run using the console @@ -61 +58,3 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or -###### To start a run + + +**To start a run** @@ -67,3 +66 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 3. On the **Runs** page, choose **Start run**. - - 4. In the **Run details** panel, provide the following information + 3. Choose **Start run**. @@ -71 +67,0 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Workflow source** \- Choose **Owned workflow** or **Shared workflow**. @@ -73 +68,0 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Workflow ID** \- The workflow ID associated with this run. @@ -75 +69,0 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Workflow version** (Optional) - Select a workflow version to use for this run. If you don't select a version, the run uses the workflow default version. @@ -77 +71 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Run name** \- A distinctive name for this run. +### Step 1: Specify run details @@ -79 +73 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Run priority** (Optional) - The priority of this run. Higher numbers specify a higher priority, and the highest priority tasks are run first. +Provide the following settings: @@ -81 +75,14 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Run storage type** \- Specify the storage type here to override the default run storage type specified for the workflow. Static storage allocates a fixed amount of storage for the run. Dynamic storage scales up and down as required for each task in the run. +# | Setting | Required | Description +---|---|---|--- +1 | **Choose workflow source** | Required | Choose **Owned workflow** (private workflows you own) or **Shared workflow** (workflows shared with you). +2 | **Workflow ID** | Required | Select the workflow ID for this run. +3 | **Run name** | Required (optional on API and CLI) | A descriptive name for this run. Maximum 127 characters. A run ID is generated automatically once the workflow is run. +4 | **Configuration** | Optional | Choose a configuration to specify VPC settings (subnets, security groups) for internet connectivity and to include container registry mappings and Git repository connections. Cannot be changed after the run starts. +5 | **Run priority** | Optional | Sets the priority of a run in a run group. A higher number means a greater priority. Integers only, in the range 0–1,000. +6 | **Run storage type** | Optional | Choose **Dynamic storage** (default, recommended) or **Static storage**. Dynamic storage scales up and down per task. Static storage provisions a fixed amount. For more information, see [Run storage types in HealthOmics workflows](./workflows-run-types.html). +7 | **Run storage capacity** | Conditional | For static storage only. Specify the amount in GiB. +8 | **Select S3 output destination** | Required | The Amazon S3 location where run outputs are delivered. Format: `s3://bucket/prefix/object`. +9 | **Output bucket owner's account ID** | Optional | If your account doesn't own the output bucket, enter the bucket owner's AWS account ID. +10 | **Run metadata retention mode** | Optional | Choose **Retain run metadata** (default) or **Remove oldest automatically**. `RETAIN` is the default value; in this mode HealthOmics doesn't delete the run metadata. For more information, see [Run retention mode for HealthOmics runs](./run-retention.html). +11 | **Network access** | Optional | Choose **Restricted** (default) or **Virtual Private Cloud (VPC)**. For more information, see VPC networking. +12 | **Service role** | Required | Choose an existing service role or create a new one. HealthOmics requires permissions for Amazon S3 and KMS. For more information, see [Service roles for AWS HealthOmics](./permissions-service.html). @@ -83 +90 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Run storage capacity** \- For static run storage, specify the amount of storage needed for the run. This entry overrides the default run storage amount specified for the workflow. +Choose **Next** to proceed to Step 2. @@ -85 +92 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Select S3 output destination** \- The S3 location where the run outputs will be saved. +### Step 2: Add parameter values @@ -87 +94 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Output bucket owner's account ID** (Optional) - If your account doesn't own the output bucket, enter the bucket owner's AWS account ID. This information is required so that HealthOmics can verify the bucket ownership. +On this page, enter the parameter values for the run, or select predefined values provided by the workflow author. @@ -89 +96 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - * **Run metadata retention mode** \- Choose whether to retain the metadata for all runs or have the system remove the oldest run metadata when your account reaches the maximum number of runs. For more information, see [Run retention mode for HealthOmics runs](./run-retention.html). +When you select a Nextflow workflow to start the run, additional sections for **Nextflow engine settings** and **Nextflow profiles** appear at the top of this step. For complete details on these settings (supported values, behavior, and profile precedence), see Specify Nextflow engine settings. @@ -91 +98 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 5. Under **Service role** , you can use an existing service role or create a new one. +**Run parameter values** @@ -93 +100 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 6. (Optional) For **Tags** , you can assign up to 50 tags to the run. +Provide the run parameters. You can upload a JSON file or manually enter the values. The JSON file contains the exact name of each input parameter and a value for the parameter. @@ -95 +102 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 7. Choose **Next**. +HealthOmics supports the following JSON types for parameter values. @@ -97 +104,8 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 8. On the **Add parameter values** page, provide the run parameters. You can either upload a JSON file that specifies the parameters or manually enter the values. +JSON type | Example key and value | Notes +---|---|--- +boolean | "b":true | Value is not in quotes, and all lowercase. +integer | "i":7 | Value is not in quotes. +number | "f":42.3 | Value is not in quotes. +string | "s":"characters" | Value is in quotes. Use string type for text values and URIs. The URI target must be the expected input type. +array | "a":[1,2,3] | Value is not in quotes. Array members must each have the type defined by the input parameter. +object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or Struct @@ -99 +113 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 9. Choose **Next**. +For more information, see [HealthOmics run inputs](./workflows-run-inputs.html) and [Managing run parameters size](./workflows-run-inputs.html#run-input-file-options). @@ -101 +115 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 10. In the **Run group** panel, you can optionally specify a run group for this run. For more information, see [Using HealthOmics run groups](./creating-run-groups.html). +Choose **Next** to proceed to Step 3. @@ -103 +117 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 11. In the **Run cache** panel, you can optionally specify a run cache for this run. For more information, see [Configuring a run with run cache using the console](./workflow-cache-startrun.html#workflow-cache-startrun-console). +### Step 3: Add run group, run cache, and tags @@ -105 +119 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 12. Choose **Review and start run**. +All settings on this page are optional. @@ -107 +121,5 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or - 13. After you review the run configuration, choose **Start run**. +# | Setting | Required | Description +---|---|---|--- +1 | **Run group** | Optional | Select an existing run group or create a new one. Run groups bundle runs by category and priority and set maximum vCPUs and run time. For more information, see [Using HealthOmics run groups](./creating-run-groups.html). +2 | **Run cache** | Optional | Use a run cache to reuse completed task results rather than recomputing them. For more information, see [Configuring a run with run cache using the console](./workflow-cache-startrun.html#workflow-cache-startrun-console). +3 | **Tags** | Optional | Add up to 50 key-value tags for search, filtering, and cost tracking. @@ -108,0 +127 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or +Choose **Next** to proceed to Step 4. @@ -109,0 +129 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or +### Step 4: Review and start run @@ -110,0 +131 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or +Review the run configuration from all previous steps. To modify a setting, choose **Edit** next to the relevant step. When you're ready, choose **Start run**. @@ -114 +135,3 @@ object | "o":{"left":"a", "right":1} | In WDL, object maps to WDL Pair, Map, or -Use the **start-run** API operation to create and start a run. +Use the **StartRun** API operation to create and start a run. + +### Start a basic run @@ -116 +139 @@ Use the **start-run** API operation to create and start a run. -The following example specifies the workflow ID and service role. This example sets the retention mode to `REMOVE`. For more information about retention mode, see [Run retention mode for HealthOmics runs](./run-retention.html). +The following example specifies the workflow ID, service role, and output URI. This example sets the retention mode to `REMOVE`. For more information about retention mode, see [Run retention mode for HealthOmics runs](./run-retention.html). @@ -119 +142 @@ The following example specifies the workflow ID and service role. This example s - aws omics start-run + aws omics start-run \ @@ -121,2 +144,3 @@ The following example specifies the workflow ID and service role. This example s - --role-arn arn:aws:iam::1234567892012:role/service-role/OmicsWorkflow-20221004T164236 \ - --name workflow name \ + --role-arn arn:aws:iam::123456789012:role/OmicsRole \ + --output-uri s3://amzn-s3-demo-bucket/output \ + --name "my-workflow-run" \ @@ -129 +153 @@ In response, you get the following output. The `uuid` is unique to the run, and - "arn": "arn:aws:omics:us-west-2:....:run/1234567", + "arn": "arn:aws:omics:us-west-2:123456789012:run/1234567", @@ -132 +156 @@ In response, you get the following output. The `uuid` is unique to the run, and - "outputUri":"s3://bucket/folder/8405154/96c57683-74bf-9d6d-ae7e-f09b097db14a" + "outputUri": "s3://bucket/folder/8405154/96c57683-74bf-9d6d-ae7e-f09b097db14a", @@ -136 +160,3 @@ In response, you get the following output. The `uuid` is unique to the run, and -### Include a parameter file +### Common API options + +**Include a parameter file** @@ -140 +166 @@ If the parameter template for a workflow declares any required parameters, you c -Reference the input JSON file in the AWS CLI by adding `--parameters file://<input_file.json>` to your `start-run` request. For more information about run parameters, see [HealthOmics run inputs](./workflows-run-inputs.html). +Reference the input JSON file in the AWS CLI by adding `--parameters file://<input_file.json>` to your `start-run` request. For more information, see supported JSON types for parameter values in Step 2: Add parameter values and [HealthOmics run inputs](./workflows-run-inputs.html). @@ -142 +168 @@ Reference the input JSON file in the AWS CLI by adding `--parameters file://<inp -### Provide a request ID +**Provide a request ID** @@ -146 +172,2 @@ You can provide a unique `requestId` for each run. The request ID is an idempote -If you use infrastructure (such as Lambda functions or step functions) for orchestrating run starts, best practice is to provide a unique request ID for each StartRun request. This ensures that if your infrastructure inadvertently starts a run that it already started, HealthOmics won't start the duplicate run. For example, if the infrastructure is attemping to recover from an upstream error, it may rerun a script that tries to start runs that are duplicate requests.