AWS omics documentation change
Summary
Restructured documentation to focus on run output summaries in JSON format rather than task-level outputs. Removed Nextflow-specific content and detailed WDL/CWL task output examples. Added JSON schema examples for output summaries and clarified how to use them for output validation.
Security assessment
The changes are documentation restructuring and clarification about output formats, with no mention of security vulnerabilities, access controls, or encryption. The modifications focus on improving output visibility and programmatic access rather than addressing security concerns.
Diff
diff --git a/omics/latest/dev/workflows-run-outputs.md b/omics/latest/dev/workflows-run-outputs.md index 1667263f5..796e0645d 100644 --- a//omics/latest/dev/workflows-run-outputs.md +++ b//omics/latest/dev/workflows-run-outputs.md @@ -5 +5 @@ -Task outputs for WDLTask outputs for NextflowTask outputs for CWL +Run output summary for WDLRun output summary for CWL @@ -7 +7 @@ Task outputs for WDLTask outputs for NextflowTask outputs for CWL -# Task outputs in a HealthOmics workflow definition +# HealthOmics run outputs @@ -9 +9 @@ Task outputs for WDLTask outputs for NextflowTask outputs for CWL -You specify task outputs in the workflow definition. By default, HealthOmics discards all intermediate task files when the workflow completes. To export an intermediate file, you define it as an output. +When a WDL or CWL run completes, the outputs include an output summary file (in JSON format) that lists all the outputs produced by the run. You can use the output summary file for these purposes: @@ -11,11 +11 @@ You specify task outputs in the workflow definition. By default, HealthOmics dis -If you use call caching, HealthOmics saves task outputs to the cache, including any intermediate files that you define as outputs. - -The following topics include task definition examples for each of the workflow definition languages. - -###### Topics - - * Task outputs for WDL - - * Task outputs for Nextflow - - * Task outputs for CWL + * Programmatically determine the output files that the run generated. @@ -22,0 +13 @@ The following topics include task definition examples for each of the workflow d + * Validate that the run produced all the expected outputs. @@ -26,3 +16,0 @@ The following topics include task definition examples for each of the workflow d -## Task outputs for WDL - -For workflow definitions written in WDL, define your outputs in the top level workflow **outputs** section. @@ -32,9 +20 @@ For workflow definitions written in WDL, define your outputs in the top level wo - * Task output for STDOUT - - * Task output for STDERR - - * Task output to a file - - * Task output to an array of files - - + * Run output summary for WDL @@ -41,0 +22 @@ For workflow definitions written in WDL, define your outputs in the top level wo + * Run output summary for CWL @@ -43 +23,0 @@ For workflow definitions written in WDL, define your outputs in the top level wo -### Task output for STDOUT @@ -45 +24,0 @@ For workflow definitions written in WDL, define your outputs in the top level wo -This example creates a task named `SayHello` that echoes the STDOUT content to the task output file. The WDL **stdout** function captures the STDOUT content (in this example, the input string **Hello World!**) in file **stdout_file**. @@ -47 +25,0 @@ This example creates a task named `SayHello` that echoes the STDOUT content to t -Because HealthOmics creates logs for all STDOUT content, the output also appears in CloudWatch Logs, along with other STDERR logging information for the task. @@ -48,0 +27 @@ Because HealthOmics creates logs for all STDOUT content, the output also appears +## Run output summary for WDL @@ -50,88 +29 @@ Because HealthOmics creates logs for all STDOUT content, the output also appears - version 1.0 - workflow HelloWorld { - input { - String message = "Hello, World!" - String ubuntu_container = "123456789012.dkr.ecr.us-east-1.amazonaws.com/dockerhub/library/ubuntu:20.04" - } - - call SayHello { - input: - message = message, - container = ubuntu_container - } - - output { - File stdout_file = SayHello.stdout_file - } - } - - task SayHello { - input { - String message - String container - } - - command <<< - echo "~{message}" - echo "Current date: $(date)" - echo "This message was printed to STDOUT" - >>> - - runtime { - docker: container - cpu: 1 - memory: "2 GB" - } - - output { - File stdout_file = stdout() - } - } - -### Task output for STDERR - -This example creates a task named `SayHello` that echoes the STDERR content to the task output file. The WDL **stderr** function captures the STDERR content (in this example, the input string **Hello World!**) in file **stderr_file**. - -Because HealthOmics creates logs for all STDERR content, the output will appear in CloudWatch Logs, along with other STDERR logging information for the task. - - - version 1.0 - workflow HelloWorld { - input { - String message = "Hello, World!" - String ubuntu_container = "123456789012.dkr.ecr.us-east-1.amazonaws.com/dockerhub/library/ubuntu:20.04" - } - - call SayHello { - input: - message = message, - container = ubuntu_container - } - - output { - File stderr_file = SayHello.stderr_file - } - } - - task SayHello { - input { - String message - String container - } - - command <<< - echo "~{message}" >&2 - echo "Current date: $(date)" >&2 - echo "This message was printed to STDERR" >&2 - >>> - - runtime { - docker: container - cpu: 1 - memory: "2 GB" - } - - output { - File stderr_file = stderr() - } - } +When a WDL run completes, HealthOmics creates an output summary file named **output.json**. @@ -139 +31 @@ Because HealthOmics creates logs for all STDERR content, the output will appear -### Task output to a file +For each output of the workflow, there is a corresponding key/value pair in the file. The key contains the workflow name and output name in the following format: `WorkflowName.output_name`. For a file output, the value is an S3 URI pointing to the output location in S3 where the file is stored. For an Array[File] output, the value is an array of S3 URIs. @@ -141 +33 @@ Because HealthOmics creates logs for all STDERR content, the output will appear -In this example, the SayHello task creates two files (message.txt and info.txt) and explicitly declares these files as the named outputs (message_file and info_file). +The following example shows the **output.json** file for a workflow named **BWAMappingWorkflow**. @@ -144,101 +36,20 @@ In this example, the SayHello task creates two files (message.txt and info.txt) - version 1.0 - workflow HelloWorld { - input { - String message = "Hello, World!" - String ubuntu_container = "123456789012.dkr.ecr.us-east-1.amazonaws.com/dockerhub/library/ubuntu:20.04" - } - - call SayHello { - input: - message = message, - container = ubuntu_container - } - - output { - File message_file = SayHello.message_file - File info_file = SayHello.info_file - } - } - - task SayHello { - input { - String message - String container - } - - command <<< - # Create message file - echo "~{message}" > message.txt - - # Create info file with date and additional information - echo "Current date: $(date)" > info.txt - echo "This message was saved to a file" >> info.txt - >>> - - runtime { - docker: container - cpu: 1 - memory: "2 GB" - } - - output { - File message_file = "message.txt" - File info_file = "info.txt" - } - } - -### Task output to an array of files - -In this example, the `GenerateGreetings` task generates an array of files as the task output. The task dynamically generates one greeting file for each member of the input array `names`. Because the file names are not known until runtime, the output definition uses the WDL glob() function to output all files that match the pattern `*_greeting.txt`. -