Merge pull request #243 from common-workflow-language/mrc-edits

[WIP] edits

Author: mr-c

draft-4/CommandLineTool.yml

@@ -41,9 +41,16 @@ $graph:
       This specification represents the fourth milestone of the CWL group.
       Since draft-3, this draft introduces the following changes and additions:
 
-        * Preliminary support for initialWorkDir & the Directory type
+        * The [Directory](#Directory) type.
         * "id: name" is now just "name"; "class: Classname" is now just
-          "Classtname".
+          "Classname".
+        * [InitialWorkDirRequirement](#InitialWorkDirRequirement): list of
+          files and subdirectories to be present in the output directory prior
+          to execution.
+        * Shortcuts for specifying the standard [output](#stdout) and/or
+          [error](#stderr) streams as a (streamable) File output.
+        * SoftwareRequirement: a lightweight method of describing software
+          dependencies
 
       ## Purpose
 
@@ -385,15 +392,33 @@ $graph:
     is equivalent to
     ```
     outputs:
-       an_output_name:
-         type: File
-         outputBinding:
-           glob: a_stdout_file
+      an_output_name:
+        type: File
+        streamable: true
+        outputBinding:
+          glob: a_stdout_file
 
     stdout: a_stdout_file
     ```
 
     If there is no `stdout` name provided, a random filename will be created.
+    For example, the following
+    ```
+    outputs:
+      an_output_name:
+        type: stdout
+    ```
+    is equivalent to
+    ```
+    outputs:
+      an_output_name:
+        type: File
+        streamable: true
+        outputBinding:
+          glob: random_stdout_filenameABCDEFG
+
+    stdout: random_stdout_filenameABCDEFG
+    ```
 
 
 - name: stderr
@@ -407,21 +432,42 @@ $graph:
     The following
     ```
     outputs:
-       an_output_name:
-       type: stderr
+      an_output_name:
+      type: stderr
 
     stderr: a_stderr_file
     ```
     is equivalent to
     ```
     outputs:
-       an_output_name:
-         type: File
-         outputBinding:
-           glob: a_stderr_file
+      an_output_name:
+        type: File
+        streamable: true
+        outputBinding:
+          glob: a_stderr_file
+
+    stderr: a_stderr_file
     ```
 
     If there is no `stderr` name provided, a random filename will be created.
+    For example, the following
+    ```
+    outputs:
+      an_output_name:
+        type: stderr
+    ```
+    is equivalent to
+    ```
+    outputs:
+      an_output_name:
+        type: File
+        streamable: true
+        outputBinding:
+          glob: random_stderr_filenameABCDEFG
+
+    stderr: random_stderr_filenameABCDEFG
+    ```
+
 
 - type: record
   name: CommandLineTool

draft-4/Process.yml

@@ -67,10 +67,10 @@ $graph:
     - name: location
       type: string?
       doc: |
-        A URI that identifies the file resource.  This may be a relative
-        reference, in which case it must be resolved using the base URI of the
+        An IRI that identifies the file resource.  This may be a relative
+        reference, in which case it must be resolved using the base IRI of the
         document.  The location may refer to a local or remote resource; the
-        implementation must use the URI to retrieve file content.  If an
+        implementation must use the IRI to retrieve file content.  If an
         implementation is unable to retrieve the file content stored at a
         remote resource (due to unsupported protocol, access denied, or other
         issue) it must signal an error.
@@ -98,6 +98,14 @@ $graph:
         As a special case, if the `path` field is provided but the `location`
         field is not, an implementation may assign the value of the `path`
         field to `location`, and remove the `path` field.
+        
+        If the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
+        (`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\`, `"`, `'`,
+        `<space>`, `<tab>`, and `<newline>`) or characters
+        [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+        for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+        then implementations may terminate the process with a
+        `permanentFailure`.
       jsonldPredicate:
         "_id": "cwl:path"
         "_type": "@id"
@@ -175,7 +183,7 @@ $graph:
         _type: "@id"
         identity: true
       doc: |
-        The format of the file.  This must be a URI of a concept node that
+        The format of the file.  This must be a IRI of a concept node that
         represents the file format, preferrably defined within an ontology.
         If no ontology is available, file formats may be tested by exact match.
 
@@ -223,11 +231,11 @@ $graph:
     - name: location
       type: string?
       doc: |
-        A URI that identifies the directory resource.  This may be a relative
-        reference, in which case it must be resolved using the base URI of the
+        An IRI that identifies the directory resource.  This may be a relative
+        reference, in which case it must be resolved using the base IRI of the
         document.  The location may refer to a local or remote resource.  If
         the `listing` field is not set, the implementation must use the
-        location URI to retrieve directory listing.  If an implementation is
+        location IRI to retrieve directory listing.  If an implementation is
         unable to retrieve the directory listing stored at a remote resource (due to
         unsupported protocol, access denied, or other issue) it must signal an
         error.
@@ -250,6 +258,14 @@ $graph:
         must not be used in any other context.  The command line tool being
         executed must be able to to access the directory at `path` using the POSIX
         `opendir(2)` syscall.
+        
+        If the `path` contains [POSIX shell metacharacters](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
+        (`|`,`&`, `;`, `<`, `>`, `(`,`)`, `$`,`` ` ``, `\`, `"`, `'`,
+        `<space>`, `<tab>`, and `<newline>`) or characters
+        [not allowed](http://www.iana.org/assignments/idna-tables-6.3.0/idna-tables-6.3.0.xhtml)
+        for [Internationalized Domain Names for Applications](https://tools.ietf.org/html/rfc6452)
+        then implementations may terminate the process with a
+        `permanentFailure`.
       jsonldPredicate:
         _id: "cwl:path"
         _type: "@id"
@@ -349,7 +365,7 @@ $graph:
       doc: |
         Only valid when `type: File` or is an array of `items: File`.
 
-        For input parameters, this must be one or more URIs of a concept nodes
+        For input parameters, this must be one or more IRIs of a concept nodes
         that represents file formats which are allowed as input to this
         parameter, preferrably defined within an ontology.  If no ontology is
         available, file formats may be tested by exact match.
@@ -683,8 +699,7 @@ $graph:
       doc: "A long, human-readable description of this process object."
     - name: cwlVersion
       type:
-        - "null"
-        - "#CWLVersions"
+        - CWLVersions
       doc: "CWL document version"
       jsonldPredicate:
         "_id": "cwl:cwlVersion"
@@ -715,7 +730,7 @@ $graph:
   doc: |
     This field consists of an array of type definitions which must be used when
     interpreting the `inputs` and `outputs` fields.  When a `type` field
-    contain a URI, the implementation must check if the type is defined in
+    contain a IRI, the implementation must check if the type is defined in
     `schemaDefs` and use that definition.  If the type is not found in
     `schemaDefs`, it is an error.  The entries in `schemaDefs` must be
     processed in the order listed such that later schema definitions may refer

draft-4/Workflow.yml

@@ -22,11 +22,12 @@ $graph:
     - |
       # Abstract
 
-      A Workflow is an analysis task represented by a directed graph describing
-      a sequence of operations that transform an input data set to output.
-      This specification defines the Common Workflow Language (CWL) Workflow
-      description, a vendor-neutral standard for representing workflows
-      intended to be portable across a variety of computing platforms.
+      One way to define a workflow is: an analysis task represented by a
+      directed graph describing a sequence of operations that transform an
+      input data set to output. This specification defines the Common Workflow
+      Language (CWL) Workflow description, a vendor-neutral standard for
+      representing workflows intended to be portable across a variety of
+      computing platforms.
 
     - {$include: intro.md}
 
@@ -85,7 +86,7 @@ $graph:
       specializeTo: "#ExpressionToolOutputParameter"
   documentRoot: true
   doc: |
-    Execute an expression as a process step.
+    Execute an expression as a Workflow step.
   fields:
     - name: "class"
       jsonldPredicate:
@@ -164,7 +165,7 @@ $graph:
     - name: source
       doc: |
         Specifies one or more workflow parameters that will provide input to
-        the underlying process parameter.
+        the underlying step parameter.
       jsonldPredicate:
         "_id": "cwl:source"
         "_type": "@id"
@@ -187,7 +188,7 @@ $graph:
   doc: |
     The input of a workflow step connects an upstream parameter (from the
     workflow inputs, or the outputs of other workflows steps) with the input
-    parameters of the underlying process.
+    parameters of the underlying step.
 
     ## Input object
 
@@ -296,9 +297,9 @@ $graph:
   docParent: "#Workflow"
   doc: |
     A workflow step is an executable element of a workflow.  It specifies the
-    underlying process implementation (such as `CommandLineTool`) in the `run`
-    field and connects the input and output parameters of the underlying
-    process to workflow parameters.
+    underlying process implementation (such as `CommandLineTool` or another
+    `Workflow`) in the `run` field and connects the input and output parameters
+    of the underlying process to workflow parameters.
 
     # Scatter/gather
 
@@ -436,15 +437,15 @@ $graph:
     specializeTo: "#WorkflowOutputParameter"
   doc: |
     A workflow describes a set of **steps** and the **dependencies** between
-    those processes.  When a process produces output that will be consumed by a
-    second process, the first process is a dependency of the second process.
+    those steps.  When a step produces output that will be consumed by a
+    second step, the first step is a dependency of the second step.
 
     When there is a dependency, the workflow engine must execute the preceeding
-    process and wait for it to successfully produce output before executing the
-    dependent process.  If two processes are defined in the workflow graph that
-    are not directly or indirectly dependent, these processes are
-    **independent**, and may execute in any order or execute concurrently.  A
-    workflow is complete when all steps have been executed.
+    step and wait for it to successfully produce output before executing the
+    dependent step.  If two steps are defined in the workflow graph that
+    are not directly or indirectly dependent, these steps are **independent**,
+    and may execute in any order or execute concurrently.  A workflow is
+    complete when all steps have been executed.
 
     Dependencies between parameters are expressed using the `source` field on
     [workflow step input parameters](#WorkflowStepInput) and [workflow output
@@ -458,21 +459,21 @@ $graph:
 
     ## Workflow success and failure
 
-    A completed process must result in one of `success`, `temporaryFailure` or
-    `permanentFailure` states.  An implementation may choose to retry a process
+    A completed step must result in one of `success`, `temporaryFailure` or
+    `permanentFailure` states.  An implementation may choose to retry a step
     execution which resulted in `temporaryFailure`.  An implementation may
     choose to either continue running other steps of a workflow, or terminate
     immediately upon `permanentFailure`.
 
-    * If any step of a workflow execution results in `permanentFailure`, then the
-    workflow status is `permanentFailure`.
+    * If any step of a workflow execution results in `permanentFailure`, then
+    the workflow status is `permanentFailure`.
 
     * If one or more steps result in `temporaryFailure` and all other steps
     complete `success` or are not executed, then the workflow status is
     `temporaryFailure`.
 
-    * If all workflow steps are executed and complete with `success`, then the workflow
-    status is `success`.
+    * If all workflow steps are executed and complete with `success`, then the
+    workflow status is `success`.
 
     # Extensions