AWS omics documentation change
Summary
Updated documentation to clarify parameter template generation process for Nextflow/CWL/WDL workflows, added detailed parsing logic for Nextflow schema/config files, and restructured content with new sections about nested parameters and interpolation examples
Security assessment
Changes focus on workflow parameter template generation mechanics and parsing requirements without mentioning security controls, vulnerabilities, or access management. The updates are operational/documentation improvements rather than security fixes.
Diff
diff --git a/omics/latest/dev/parameter-templates.md b/omics/latest/dev/parameter-templates.md index 9cca8b598..b2e8a75ba 100644 --- a//omics/latest/dev/parameter-templates.md +++ b//omics/latest/dev/parameter-templates.md @@ -5 +5 @@ -Parameter detection for CWL and WDL workflows +Generating parameter templates @@ -9 +9 @@ Parameter detection for CWL and WDL workflows -Parameter templates define the input parameters for a workflow. You can define input parameters to make your workflow more flexible and versatile. For example, you can define a parameter to be the Amazon S3 location of the reference genome files. Users can then run the workflow using various data sets. +Parameter templates define the input parameters for a workflow. You can define input parameters to make your workflow more flexible and versatile. For example, you can define a parameter for the Amazon S3 location of the reference genome files. Users can then run the workflow using various data sets. @@ -11 +11 @@ Parameter templates define the input parameters for a workflow. You can define i -Create a parameter template for Nextflow workflows, and optionally for WDL and CWL workflows. +You can create the parameter template for your workflow, or HealthOmics can generate the parameter template for you. @@ -13 +13,3 @@ Create a parameter template for Nextflow workflows, and optionally for WDL and C -To define the input parameters, create a parameter template JSON file. In the file, each input parameter is a named object that must match the name of the workflow input. The input parameter object includes the following attributes: +The parameter template is a JSON file. In the file, each input parameter is a named object that must match the name of the workflow input. When you start a run, if you don't provide values for all the required parameters, the run fails. + +The input parameter object includes the following attributes: @@ -39 +41,9 @@ The following example parameter template shows how to specify the input paramete -## Parameter detection for CWL and WDL workflows +## Generating parameter templates + +HealthOmics generates the parameter template by parsing the workflow definition to detect input parameters. If you provide a parameter template file for a workflow, the parameters in your file override the parameters detected in the workflow definition. + +There are slight differences between the parsing logic of the CWL, WDL, and Nextflow engines, as described in the following sections. + +###### Topics + + * Parameter detection for CWL @@ -41 +51 @@ The following example parameter template shows how to specify the input paramete -Parameter templates are optional for CWL and WDL workflows. HealthOmics parses the `main` workflow definition to detect the input parameters to use. If you provide a parameter template for a CWL or WDL workflow, the template overrides the parameters detected in the workflow definition. + * Parameter detection for WDL @@ -43 +53 @@ Parameter templates are optional for CWL and WDL workflows. HealthOmics parses t -There are slight differences between the parsing logic of the CWL and WDL engines, as described in the following sections. + * Parameter detection for Nextflow @@ -45 +55,4 @@ There are slight differences between the parsing logic of the CWL and WDL engine -### Parsing logic for the CWL workflow engine + + + +### Parameter detection for CWL @@ -124 +137 @@ Input | Example input/output | Required -### Parsing logic for the WDL workflow engine +### Parameter detection for WDL @@ -184,0 +198,325 @@ Struct sample {String a, Int y} later in inputs: Sample? mySample | +### Parameter detection for Nextflow + +For Nextflow, HealthOmics generates the parameter template by parsing the `nextflow_schema.json` file. If the workflow definition doesn't include a schema file, HealthOmics parses the main workflow definition file. + +###### Topics + + * Parsing the schema file + + * Parsing the main file + + * Nested parameters + + * Examples of Nextflow interpolation + + + + +#### Parsing the schema file + +For parsing to work correctly, make sure the schema file meets the following requirements: + + * The schema file is named `nextflow_schema.json` and is located in the same directory as the main workflow file. + + * The schema file is valid JSON as defined in either of the following schemas: + + * [json-schema.org/draft/2020-12/schema](https://json-schema.org/draft/2020-12/schema). + + * [json-schema.org/draft-07/schema](https://json-schema.org/draft-07/schema). + + + + +HealthOmics parses the `nextflow_schema.json` file to generate the parameter template: + + * Extracts all **properties** that are defined in the schema. + + * Includes the property **description** if available for the property. + + * Identifies whether each parameter is optional or required, based on the **required** field of the property. + + + + +The following example shows a definition file and the generated parameter file. + + + { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "$defs": { + "input_options": { + "title": "Input options", + "type": "object", + "required": ["input_file"], + "properties": { + "input_file": { + "type": "string", + "format": "file-path", + "pattern": "^s3://[a-z0-9.-]{3,63}(?:/\\S*)?$", + "description": "description for input_file" + }, + "input_num": { + "type": "integer", + "default": 42, + "description": "description for input_num" + } + } + }, + "output_options": { + "title": "Output options", + "type": "object", + "required": ["output_dir"], + "properties": { + "output_dir": { + "type": "string", + "format": "file-path", + "description": "description for output_dir", + } + } + } + }, + "properties": { + "ungrouped_input_bool": { + "type": "boolean", + "default": true + } + }, + "required": ["ungrouped_input_bool"], + "allOf": [ + { "$ref": "#/$defs/input_options" }, + { "$ref": "#/$defs/output_options" } + ] + } + +The generated parameter template: + + + { + "input_file": { + "description": "description for input_file", + "optional": False + }, + "input_num": { + "description": "description for input_num", + "optional": True + }, + "output_dir": { + "description": "description for output_dir", + "optional": False + }, + "ungrouped_input_bool": { + "description": None, + "optional": False + } + } + +#### Parsing the main file + +If the workflow definition doesn't include a `nextflow_schema.json` file, HealthOmics parses the main workflow definition file. + +HealthOmics analyzes the `params` expressions found in the main workflow definition file and in the `nextflow.config` file. All `params` with default values are marked as optional. + +For parsing to work correctly, note the following requirements: + + * HealthOmics parses only the main workflow definition file. To ensure all parameters are captured, we recommend that you wire all **params** through to any submodules and imported workflows. + + * The config file is optional. If you define one, name it `nextflow.config` and place it in the same directory as the main workflow definition file. + + + + +The following example shows a definition file and the generated parameter template. + + + params.input_file = "default.txt" + params.threads = 4 + params.memory = "8GB" + + workflow { + if (params.version) { + println "Using version: ${params.version}" + } + } + +The generated parameter template: + + + { + "input_file": { + "description": None, + "optional": True + }, + "threads": { + "description": None, + "optional": True