Merge branch 'master' into userguide-1.0

Author: Peter Amstutz

README.md

@@ -23,7 +23,7 @@ provides a gentle introduction to writing CWL command line tool and workflow des
 
 The current stable specification is [v1.0](http://www.commonwl.org/v1.0/):
 
-http://www.commonwl.org/draft-v1.0/
+http://www.commonwl.org/v1.0/
 
 Older drafts: [draft-1](https://github.com/common-workflow-language/common-workflow-language/tree/master/draft-1), [draft-2](http://www.commonwl.org/draft-2/), [draft-3](http://www.commonwl.org/draft-3/)
 

v1.0/CommandLineTool.yml

@@ -39,18 +39,39 @@ $graph:
       ## Introduction to v1.0
 
       This specification represents the first full release from the CWL group.
-      Since draft-3, this draft introduces the following changes and additions:
+      Since draft-3, version 1.0 introduces the following changes and additions:
 
         * The [Directory](#Directory) type.
-        * "id: name" is now just "name"; "class: Classname" is now just
-          "Classname".
+        * Syntax simplifcations: denoted by the `map<>` syntax. Example: inputs
+          contains a list of items, each with an id. Now one can specify
+          a mapping of that identifier to the corresponding
+          `CommandInputParamater`.
+          ```
+          inputs:
+           - id: one
+             type: string
+             doc: First input parameter
+           - id: two
+             type: int
+             doc: Second input parameter
+          ```
+          can be
+          ```
+          inputs:
+           one:
+            type: string
+            doc: First input parameter
+           two:
+            type: int
+            doc: Second input parameter
+          ```
         * [InitialWorkDirRequirement](#InitialWorkDirRequirement): list of
           files and subdirectories to be present in the output directory prior
           to execution.
         * Shortcuts for specifying the standard [output](#stdout) and/or
           [error](#stderr) streams as a (streamable) File output.
-        * SoftwareRequirement: a lightweight method of describing software
-          dependencies
+        * [SoftwareRequirement](#SoftwareRequirement) for describing software
+          dependencies of a tool.
 
       ## Purpose
 
@@ -60,12 +81,12 @@ $graph:
       languages and execute concurrently on multiple hosts.  However, POSIX
       does not dictate computer-readable grammar or semantics for program input
       and output, resulting in extremely heterogeneous command line grammar and
-      input/output semantics among program.  This a particular problem in
+      input/output semantics among program.  This is a particular problem in
       distributed computing (multi-node compute clusters) and virtualized
       environments (such as Docker containers) where it is often necessary to
       provision resources such as input files before executing the program.
 
-      Often this is gap is filled by hard coding program invocation and
+      Often this gap is filled by hard coding program invocation and
       implicitly assuming requirements will be met, or abstracting program
       invocation with wrapper scripts or descriptor documents.  Unfortunately,
       where these approaches are application or platform specific it creates a
@@ -203,8 +224,8 @@ $graph:
           items: string
       doc: |
         Find files relative to the output directory, using POSIX glob(3)
-        pathname matching.  If provided an array, find files that match any
-        pattern in the array.  If provided an expression, the expression must
+        pathname matching.  If an array is provided, find files that match any
+        pattern in the array.  If an expression is provided, the expression must
         return a string or an array of strings, which will then be evaluated as
         one or more glob patterns.  Must only match and return files which
         actually exist.
@@ -554,14 +575,14 @@ $graph:
       type: int[]?
       doc: |
         Exit codes that indicate the process failed due to a possibly
-        temporary condition, where excuting the process with the same
+        temporary condition, where executing the process with the same
         runtime environment and inputs may produce different results.
 
     - name: permanentFailCodes
       type: int[]?
       doc:
         Exit codes that indicate the process failed due to a permanent logic
-        error, where excuting the process with the same runtime environment and
+        error, where executing the process with the same runtime environment and
         same inputs is expected to always fail.
 
 
@@ -574,7 +595,7 @@ $graph:
     the image.
 
     If a CommandLineTool lists `DockerRequirement` under
-    `hints` or `requirements`, it may (or must) be run in the specified Docker
+    `hints` (or `requirements`), it may (or must) be run in the specified Docker
     container.
 
     The platform must first acquire or install the correct Docker image as

v1.0/Process.yml

@@ -44,7 +44,7 @@ $graph:
     - cwl:File
     - cwl:Directory
   doc:
-    - "Extends primitive types with the concept of a file as a first class type."
+    - "Extends primitive types with the concept of a file and directory as a builtin type."
     - "File: A File object"
     - "Directory: A Directory object"
 
@@ -77,7 +77,7 @@ $graph:
         remote resource (due to unsupported protocol, access denied, or other
         issue) it must signal an error.
 
-        If the `location' field is not provided, the `contents` field must be
+        If the `location` field is not provided, the `contents` field must be
         provided.  The implementation must assign a unique identifier for
         the `location` field.
 
@@ -165,7 +165,7 @@ $graph:
         "sha1$ + hexadecimal string" using the SHA-1 algorithm.
     - name: size
       type: long?
-      doc: Optional file size.
+      doc: Optional file size
     - name: "secondaryFiles"
       type:
         - "null"
@@ -186,7 +186,7 @@ $graph:
         _type: "@id"
         identity: true
       doc: |
-        The format of the file.  This must be a IRI of a concept node that
+        The format of the file: this must be an IRI of a concept node that
         represents the file format, preferrably defined within an ontology.
         If no ontology is available, file formats may be tested by exact match.
 
@@ -217,7 +217,7 @@ $graph:
 
 - name: Directory
   type: record
-  docParent: "#CWLType"
+  docAfter: "#File"
   doc: |
     Represents a directory to present to a command line tool.
   fields:
@@ -362,7 +362,7 @@ $graph:
       doc: |
         Only valid when `type: File` or is an array of `items: File`.
 
-        For input parameters, this must be one or more IRIs of a concept nodes
+        For input parameters, this must be one or more IRIs of concept nodes
         that represents file formats which are allowed as input to this
         parameter, preferrably defined within an ontology.  If no ontology is
         available, file formats may be tested by exact match.
@@ -391,8 +391,8 @@ $graph:
 - type: enum
   name: Expression
   doc: |
-    Not a real type.  Indicates that a field must allow runtime parameter
-    references.  If [InlineJavascriptRequirement](#InlineJavascriptRequirement)
+    'Expression' is not a real type.  It indicates that a field must allow
+    runtime parameter references.  If [InlineJavascriptRequirement](#InlineJavascriptRequirement)
     is declared and supported by the platform, the field must also allow
     Javascript expressions.
   symbols:

v1.0/Workflow.yml

@@ -39,6 +39,29 @@ $graph:
       Since draft-3, this draft introduces the following changes and additions:
 
         * The 'inputs' and 'outputs' fields have been renamed 'in' and 'out'.
+        * Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`
+          contains a list of items, each with an id. Now one can specify
+          a mapping of that identifier to the corresponding
+          `InputParameter`.
+          ```
+          in:
+           - id: one
+             type: string
+             doc: First input parameter
+           - id: two
+             type: int
+             doc: Second input parameter
+          ```
+          can be
+          ```
+          in:
+           one:
+            type: string
+            doc: First input parameter
+           two:
+            type: int
+            doc: Second input parameter
+          ```
 
       ## Purpose
 

v1.0/concepts.md

@@ -1,4 +1,4 @@
-## References to Other Specifications
+## References to other specifications
 
 **Javascript Object Notation (JSON)**: http://json.org
 
@@ -28,7 +28,7 @@ serve as a reference for the behavior of conforming implementations.
 The terminology used to describe CWL documents is defined in the
 Concepts section of the specification. The terms defined in the
 following list are used in building those definitions and in describing the
-actions of an CWL implementation:
+actions of a CWL implementation:
 
 **may**: Conforming CWL documents and CWL implementations are permitted but
 not required to behave as described.
@@ -68,16 +68,16 @@ A **process** is a basic unit of computation which accepts input data,
 performs some computation, and produces output data. Examples include
 CommandLineTools, Workflows, and ExpressionTools.
 
-An **input object** is an object describing the inputs to a invocation of
-process.
+An **input object** is an object describing the inputs to an invocation of
+a process.
 
-An **output object** is an object describing the output of an invocation of a
-process.
+An **output object** is an object describing the output resulting from an
+invocation of a process.
 
 An **input schema** describes the valid format (required fields, data types)
 for an input object.
 
-An **output schema** describes the valid format for a output object.
+An **output schema** describes the valid format for an output object.
 
 **Metadata** is information about workflows, tools, or input items.
 
@@ -87,7 +87,7 @@ CWL documents must consist of an object or array of objects represented using
 JSON or YAML syntax.  Upon loading, a CWL implementation must apply the
 preprocessing steps described in the
 [Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html).
-A implementation may formally validate the structure of a CWL document using
+An implementation may formally validate the structure of a CWL document using
 SALAD schemas located at
 https://github.com/common-workflow-language/common-workflow-language/tree/master/draft-4
 
@@ -112,7 +112,7 @@ Another transformation defined in Schema salad is simplification of data type de
 Type `<T>` ending with `?` should be transformed to `[<T>, "null"]`.
 Type `<T>` ending with `[]` should be transformed to `{"type": "array", "items": <T>}`
 
-## Extensions and Metadata
+## Extensions and metadata
 
 Input metadata (for example, a lab sample identifier) may be represented within
 a tool or workflow using input parameters which are explicitly propagated to
@@ -138,13 +138,13 @@ associated datatype or schema.  During execution, values are assigned to
 parameters to make the input object or output object used for concrete
 process invocation.
 
-A **command line tool** is a process characterized by the execution of a
+A **CommandLineTool** is a process characterized by the execution of a
 standalone, non-interactive program which is invoked on some input,
 produces output, and then terminates.
 
 A **workflow** is a process characterized by multiple subprocess steps,
-where step outputs are connected to the inputs of other downstream steps to
-form a directed graph, and independent steps may run concurrently.
+where step outputs are connected to the inputs of downstream steps to
+form a directed acylic graph, and independent steps may run concurrently.
 
 A **runtime environment** is the actual hardware and software environment when
 executing a command line tool.  It includes, but is not limited to, the
@@ -168,14 +168,14 @@ not covered by this specification.  Some areas that are currently out of
 scope for CWL specification but may be handled by a specific workflow
 platform include:
 
-* Data security and permissions.
+* Data security and permissions
 * Scheduling tool invocations on remote cluster or cloud compute nodes.
 * Using virtual machines or operating system containers to manage the runtime
 (except as described in [DockerRequirement](CommandLineTool.html#DockerRequirement)).
 * Using remote or distributed file systems to manage input and output files.
 * Transforming file paths.
-* Determining if a process has previously been executed, skipping it and
-reusing previous results.
+* Determining if a process has previously been executed, and if so skipping it
+and reusing previous results.
 * Pausing, resuming or checkpointing processes or workflows.
 
 Conforming CWL processes must not assume anything about the runtime
@@ -190,7 +190,7 @@ command line line tools) is as follows.
 1. Load, process and validate a CWL document, yielding a process object.
 2. Load input object.
 3. Validate the input object against the `inputs` schema for the process.
-4. Validate that process requirements are met.
+4. Validate process requirements are met.
 5. Perform any further setup required by the specific process type.
 6. Execute the process.
 7. Capture results of process execution into the output object.
@@ -276,7 +276,7 @@ be provided:
     a contextual value of 'self' is not documented for a field, it
     must be 'null'.
   * `runtime`: An object containing configuration details.  Specific to the
-    process type.  An implementation may provide may provide
+    process type.  An implementation may provide
     opaque strings for any or all fields of `runtime`.  These must be
     filled in by the platform after processing the Tool but before actual
     execution.  Parameter references and expressions may only use the
@@ -288,25 +288,25 @@ characters around a parameter reference, the effective value of the field
 becomes the value of the referenced parameter, preserving the return type.
 
 If the value of a field has non-whitespace leading or trailing characters
-around an parameter reference, it is subject to string interpolation.  The
-effective value of the field is a string containing the leading characters;
-followed by the string value of the parameter reference; followed by the
+around a parameter reference, it is subject to string interpolation.  The
+effective value of the field is a string containing the leading characters,
+followed by the string value of the parameter reference, followed by the
 trailing characters.  The string value of the parameter reference is its
 textual JSON representation with the following rules:
 
   * Leading and trailing quotes are stripped from strings
   * Objects entries are sorted by key
 
-Multiple parameter references may appear in a single field.  This case is
+Multiple parameter references may appear in a single field.  This case
 must be treated as a string interpolation.  After interpolating the first
 parameter reference, interpolation must be recursively applied to the
 trailing characters to yield the final string value.
 
 ## Expressions
 
 An expression is a fragment of [Javascript/ECMAScript
-5.1](http://www.ecma-international.org/ecma-262/5.1/) code which is
-evaluated by the workflow platform to affect the inputs, outputs, or
+5.1](http://www.ecma-international.org/ecma-262/5.1/) code evaluated by the
+workflow platform to affect the inputs, outputs, or
 behavior of a process.  In the generic execution sequence, expressions may
 be evaluated during step 5 (process setup), step 6 (execute process),
 and/or step 7 (capture output).  Expressions are distinct from regular
@@ -370,7 +370,7 @@ platform's CWL implementation.
 
 A CWL input object document may similarly begin with `#!/usr/bin/env
 cwl-runner` and be marked as executable.  In this case, the input object
-must include the field `cwl:tool` supplying a IRI to the default CWL
+must include the field `cwl:tool` supplying an IRI to the default CWL
 document that should be executed using the fields of the input object as
 input parameters.
 

v1.0/contrib.md

@@ -1,15 +1,19 @@
 Authors:
 
 * Peter Amstutz <[email protected]>, Arvados Project, Curoverse
+* Michael R. Crusoe <[email protected]>, Common Workflow Language
+  project
 * Nebojša Tijanić <[email protected]>, Seven Bridges Genomics
 
 Contributers:
 
 * Brad Chapman <[email protected]>, Harvard Chan School of Public Health
 * John Chilton <[email protected]>, Galaxy Project, Pennsylvania State University
-* Michael R. Crusoe <crusoe@ucdavis.edu>, University of California, Davis
+* Michael Heuer <heuermh@berkeley.edu,>,UC Berkeley AMPLab
 * Andrey Kartashov <[email protected]>, Cincinnati Children's Hospital
 * Dan Leehr <[email protected]>, Duke University
 * Hervé Ménager <[email protected]>, Institut Pasteur
+* Maya Nedeljkovich <[email protected]>, Seven Bridges Genomics
+* Matt Scales <[email protected]>, Institute of Cancer Research, London
 * Stian Soiland-Reyes [[email protected]](mailto:[email protected]), University of Manchester
 * Luka Stojanovic <[email protected]>, Seven Bridges Genomics

v1.0/intro.md

@@ -1,4 +1,4 @@
-# Status of This Document
+# Status of this document
 
 This document is the product of the [Common Workflow Language working
 group](https://groups.google.com/forum/#!forum/common-workflow-language).  The