Merge branch 'master' of https://github.com/common-workflow-language/common-workflow-language

Author: Luka Stojanovic

README.md

@@ -1,7 +1,7 @@
 Common Workflow Language
 ========================
 
-[![GitHub stars](https://img.shields.io/github/stars/common-workflow-language/common-workflow-language.svg)](https://github.com/common-workflow-language/common-workflow-language/stargazers) [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/common-workflow-language/common-workflow-language?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![GitHub stars](https://img.shields.io/github/stars/common-workflow-language/common-workflow-language.svg)](https://github.com/common-workflow-language/common-workflow-language/stargazers) [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/common-workflow-language/common-workflow-language?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [Support](#support-community-and-contributing)
 
 The Common Workflow Language (CWL) is a specification for describing analysis
 workflows and tools in a way that makes them portable and scalable across a
@@ -12,7 +12,9 @@ Imaging, Astronomy, Physics, and Chemistry.
 
 CWL is developed by an informal, multi-vendor working group consisting of
 organizations and individuals aiming to enable scientists to share data
-analysis workflows.  [The CWL project is on Github.](https://github.com/common-workflow-language/common-workflow-language)
+analysis workflows.  [The CWL project is on Github](https://github.com/common-workflow-language/common-workflow-language) and we follow the [Open-Stand.org principles for collaborative open standards development](https://open-stand.org/about-us/principles/)
+
+[![Open Stand badge](http://standards.ieee.org/images/openstand/128x128-blue2.png)](https://open-stand.org/about-us/principles/).
 
 CWL builds on technologies such as [JSON-LD](http://json-ld.org)
 for data modeling and [Docker](http://docker.com) for portable runtime
@@ -44,8 +46,9 @@ Please cite https://dx.doi.org/10.6084/m9.figshare.3115156.v2
 |--------|-----------|-----------|--------|
 |[cwltool](https://github.com/common-workflow-language/cwltool)|Reference implementation of CWL|[![Build Status](https://ci.commonwl.org/job/cwltool-conformance/badge/icon)](http://ci.commonwl.org/job/cwltool-conformance/lastBuild/testReport/)|Linux, OS X, local execution only|
 |[Arvados](https://arvados.org)|Distributed computing platform for data analysis on massive data sets. [Using CWL on Arvados](http://doc.arvados.org/user/cwl/cwl-runner.html)|[![Build Status](https://ci.commonwl.org/job/arvados-conformance/badge/icon)](http://ci.commonwl.org/job/arvados-conformance/lastBuild/testReport/)|AWS, GCP, Azure, Slurm|
-|[Toil](https://github.com/BD2KGenomics/toil)|Toil is a workflow engine entirely written in Python.|[![Build Status](https://ci.commonwl.org/job/toil-conformance/badge/icon)](http://ci.commonwl.org/job/toil-conformance/lastBuild/testReport/)|AWS, GCP, Azure, OpenStack, Grid Engine, Mesos|
-|[Rabix Bunny](https://github.com/rabix/bunny)|Reproducible Analyses for Bioinformatics|[![Build Status](https://ci.commonwl.org/buildStatus/icon?job=rabix-conformance)](https://ci.commonwl.org/job/rabix-conformance/)|Java|
+|[Toil](https://github.com/BD2KGenomics/toil)|Toil is a workflow engine entirely written in Python.|[![Build Status](https://ci.commonwl.org/job/toil-conformance/badge/icon)](http://ci.commonwl.org/job/toil-conformance/lastBuild/testReport/)|AWS, GCP, Azure, Slurm, OpenStack, Grid Engine, Mesos|
+|[Rabix Bunny](https://github.com/rabix/bunny)|An open-source, Java-based implementation of Common Workflow Language with support for multiple drafts/versions. See [Rabix.io](http://rabix.io) for details.|[![Build Status](https://ci.commonwl.org/buildStatus/icon?job=rabix-conformance)](https://ci.commonwl.org/job/rabix-conformance/)|Linux, OS X, [GA4GH TES](https://github.com/ga4gh/task-execution-server) (experimental)|
+|[Consonance](https://github.com/Consonance/consonance)|orchestration tool for running SeqWare workflows and CWL tools|[pending](https://ci.commonwl.org/job/rabix-conformance/)|AWS, OpenStack, Azure|
 |[Apache Taverna](http://taverna.incubator.apache.org/)|Domain-independent Workflow Management System|[alpha](https://issues.apache.org/jira/browse/TAVERNA-900)|Java|
 |[Galaxy](https://galaxyproject.org/)|Web-based platform for data intensive biomedical research.|[alpha](https://github.com/common-workflow-language/galaxy)|Python|
 |[AWE](https://github.com/MG-RAST/AWE)|Workflow and resource management system for bioinformatics data analysis.|[alpha](https://github.com/wgerlach/AWE)|Go|
@@ -73,6 +76,16 @@ Please cite https://dx.doi.org/10.6084/m9.figshare.3115156.v2
 |[cwlavro](https://github.com/common-workflow-language/cwlavro)|Java classes for loading CWL documents|
 |[acd2cwl](https://github.com/common-workflow-language/acd2cwl)|CWL generator for ACD (EMBOSS) files |
 |[CWLviewer](https://view.commonwl.org/)|A web application to view and share Common Workflow Language workflows|
+|[scriptcwl](https://github.com/NLeSC/scriptcwl)|Create CWL workflows by writing a simple Python script|
+|[cwl2zshcomp](https://github.com/kloetzl/cwl2zshcomp)|generates ZSH auto completions from CWL command line tool descriptions|
+
+## Projects the CWL community is participating in
+
+|Name|Details|
+|-------|-----------|
+|[GA4GH Task Execution API](https://github.com/ga4gh/task-execution-schemas/)|a minimal common API for submitting a single job to a remote execution endpoint. Many contributions from CWL project participants.|
+|[GA4GH Workflow Execution API](https://github.com/ga4gh/workflow-execution-schemas)|a minimal common API for submitting workflow requests to workflow execution systems in a standardized way. Many contributions from CWL project participants.|
+|[Bio-compute objects](https://hive.biochemistry.gwu.edu/htscsrs?pageid=biocompute)|"a step towards evaluation and validation of bio-medical scientific computations", CWL and researchobject.org participants are cooperating with this effort|
 
 ## Support, Community and Contributing
 
@@ -115,12 +128,13 @@ Conduct](https://github.com/common-workflow-language/common-workflow-language/bl
 * [Institut Pasteur](http://www.pasteur.fr)
 * [Wellcome Trust Sanger Institute](https://www.sanger.ac.uk/)
 * [University of California Santa Cruz](https://cbse.soe.ucsc.edu/research/bioinfo)
-* [Harvard Chan School of Public Health](http://www.hsph.harvard.edu/)
+* [Harvard T.H. Chan School of Public Health](http://www.hsph.harvard.edu/)
 * [Cincinnati Children's Hospital Medical Center](http://www.cincinnatichildrens.org/)
 * [Broad Institute](https://www.broadinstitute.org)
-* [Wolfson Wohl Cancer Research Centre](http://www.gla.ac.uk/researchinstitutes/cancersciences/ics/facilities/wwcrc/)
+* [University of Melbourne Center for Cancer Research](https://umccr.github.io/)
 * [Netherlands eScience Center](https://www.esciencecenter.nl/)
-* [Texas Advanced Computing Center Life Science Computing Group](https://www.tacc.utexas.edu/life-sciences-computing)
+* [Texas Advanced Computing Center Life Science Computing Group](https://www.tacc.utexas.edu/life-sciences-computing) / [Agave Platform](https://agaveapi.co/)
+* [CyVerse](http://www.cyverse.org/)
 
 ## Individual Contributors
 

v1.0/UserGuide.yml

@@ -183,7 +183,7 @@
 
       The value of `position` is used to determine where parameter should
       appear on the command line.  Positions are relative to one another, not
-      abosolute.  As a result, positions do not have to be sequential, three
+      absolute.  As a result, positions do not have to be sequential, three
       parameters with positions `[1, 3, 5]` will result in the same command
       line as `[1, 2, 3]`.  More than one parameter can have the same position
       (ties are broken using the parameter name), and the position field itself
@@ -239,7 +239,7 @@
 
       ```
       outputs:
-        - id: example_out
+        example_out:
           type: File
           outputBinding:
             glob: hello.txt
@@ -466,7 +466,7 @@
       tool is actually executed.  The `$(runtime.outdir)` parameter is the path
       to the designated output directory.  Other parameters include
       `$(runtime.tmpdir)`, `$(runtime.ram)`, `$(runtime.cores)`,
-      `$(runtime.ram)`, `$(runtime.outdirSize)`, and `$(runtime.tmpdirSize)`.  See
+      `$(runtime.outdirSize)`, and `$(runtime.tmpdirSize)`.  See
       the [Runtime Environment](CommandLineTool.html#Runtime_environment)
       section of the CWL specification for details.
 

v1.0/conformance_test_v1.0.yaml

@@ -78,33 +78,33 @@
   doc: Test command execution in Docker with stdout redirection
 
 - job: v1.0/empty.json
-  tool: v1.0/egrep-stderr.cwl
+  tool: v1.0/stderr.cwl
   doc: Test command line with stderr redirection
   output:
     output_file:
       class: File
-      checksum: sha1$cec7b8746a78c42060c96505887449bca0142976
-      size: 84
+      checksum: sha1$f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
+      size: 4
       location: error.txt
 
 - job: v1.0/empty.json
-  tool: v1.0/egrep-stderr-shortcut.cwl
+  tool: v1.0/stderr-shortcut.cwl
   doc: Test command line with stderr redirection, brief syntax
   output:
     output_file:
       class: File
-      checksum: sha1$cec7b8746a78c42060c96505887449bca0142976
-      size: 84
+      checksum: sha1$f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
+      size: 4
       location: Any
 
 - output:
     output_file:
       class: File
-      size: 84
-      checksum: sha1$cec7b8746a78c42060c96505887449bca0142976
+      size: 4
+      checksum: sha1$f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
       location: std.err
   job: v1.0/empty.json
-  tool: v1.0/egrep-stderr-mediumcut.cwl
+  tool: v1.0/stderr-mediumcut.cwl
   doc: Test command line with stderr redirection, named brief syntax
 
 - job: v1.0/cat-job.json

v1.0/v1.0/dir5.cwl

@@ -11,7 +11,7 @@ outputs:
     type: File
     outputBinding:
       glob: output.txt
-arguments: ["find", ".",
+arguments: ["find", "-L", ".",
   {shellQuote: false, valueFrom: "|"},
   "sort"]
 stdout: output.txt

v1.0/v1.0/egrep-stderr-mediumcut.cwl renamed to v1.0/v1.0/stderr-mediumcut.cwl

@@ -2,17 +2,12 @@
 class: CommandLineTool
 cwlVersion: v1.0
 doc: "Test of capturing stderr output in a docker container."
-hints:
-  DockerRequirement:
-    dockerPull: debian:wheezy
-
+requirements:
+  ShellCommandRequirement: {}
 inputs: []
-
 outputs:
   output_file:
     type: stderr
-
-baseCommand: egrep
-successCodes: [2]
-
+arguments:
+ - { valueFrom: "echo foo 1>&2", shellQuote: False }
 stderr: std.err

v1.0/v1.0/egrep-stderr-shortcut.cwl renamed to v1.0/v1.0/stderr-shortcut.cwl

@@ -2,15 +2,11 @@
 class: CommandLineTool
 cwlVersion: v1.0
 doc: "Test of capturing stderr output in a docker container."
-hints:
-  DockerRequirement:
-    dockerPull: debian:wheezy
-
+requirements:
+  ShellCommandRequirement: {}
 inputs: []
-
 outputs:
   output_file:
     type: stderr
-
-baseCommand: egrep
-successCodes: [2]
+arguments:
+ - { valueFrom: "echo foo 1>&2", shellQuote: False }

v1.0/v1.0/egrep-stderr.cwl renamed to v1.0/v1.0/stderr.cwl

@@ -2,14 +2,13 @@
 class: CommandLineTool
 cwlVersion: v1.0
 doc: "Test of capturing stderr output in a docker container."
-hints:
-  DockerRequirement:
-    dockerPull: debian:wheezy
+requirements:
+  ShellCommandRequirement: {}
 inputs: []
 outputs:
   output_file:
     type: File
     outputBinding: {glob: error.txt}
-baseCommand: egrep
-successCodes: [2]
+arguments:
+ - { valueFrom: "echo foo 1>&2", shellQuote: False }
 stderr: error.txt

v1.1.0-dev1/CommandLineTool.yml

@@ -747,7 +747,6 @@ $graph:
     - name: package
       type: string
       doc: "The common name of the software to be configured."
-      jsonldPredicate: "@id"
     - name: version
       type: string[]?
       doc: "The (optional) version of the software to configured."

v1.1.0-dev1/Process.yml

@@ -595,7 +595,6 @@ $graph:
       doc: |
         Specify valid types of data that may be assigned to this parameter.
 
-
 - name: OutputParameter
   type: record
   extends: Parameter

v1.1.0-dev1/UserGuide.yml

@@ -183,7 +183,7 @@
 
       The value of `position` is used to determine where parameter should
       appear on the command line.  Positions are relative to one another, not
-      abosolute.  As a result, positions do not have to be sequential, three
+      absolute.  As a result, positions do not have to be sequential, three
       parameters with positions `[1, 3, 5]` will result in the same command
       line as `[1, 2, 3]`.  More than one parameter can have the same position
       (ties are broken using the parameter name), and the position field itself
@@ -239,7 +239,7 @@
 
       ```
       outputs:
-        - id: example_out
+        example_out:
           type: File
           outputBinding:
             glob: hello.txt
@@ -290,8 +290,8 @@
     - |
       ## Parameter references
 
-      In a previous example, we used extracted a file using the "tar" program.
-      However, that example was very limited becuase it assumed that the file
+      In a previous example, we extracted a file using the "tar" program.
+      However, that example was very limited because it assumed that the file
       we were interested in was called "hello.txt".  In this example, you will
       see how to reference the value of input parameters dynamically from other
       fields.
@@ -356,7 +356,7 @@
       containers are also purposefully isolated from the host system, so in
       order to run a tool inside a Docker container there is additional work to
       ensure that input files are available inside the container and output
-      files can be recovered from the contianer.  CWL can perform this work
+      files can be recovered from the container.  CWL can perform this work
       automatically, allowing you to use Docker to simplify your software
       management while avoiding the complexity of invoking and managing Docker
       containers.
@@ -466,7 +466,7 @@
       tool is actually executed.  The `$(runtime.outdir)` parameter is the path
       to the designated output directory.  Other parameters include
       `$(runtime.tmpdir)`, `$(runtime.ram)`, `$(runtime.cores)`,
-      `$(runtime.ram)`, `$(runtime.outdirSize)`, and `$(runtime.tmpdirSize)`.  See
+      `$(runtime.outdirSize)`, and `$(runtime.tmpdirSize)`.  See
       the [Runtime Environment](CommandLineTool.html#Runtime_environment)
       section of the CWL specification for details.
 
@@ -522,7 +522,7 @@
     - |
       ```
 
-      *array-outpust-job.yml*
+      *array-outputs-job.yml*
       ```
     - $include: examples/array-outputs-job.yml
     - |
@@ -662,6 +662,16 @@
     - |
       ```
 
+      As this tool does not require any `inputs` we can run it with an (almost) empty job file:
+
+      *empty.yml*
+      ```
+      {}
+      |
+      ```
+
+      We can then run `expression.cwl`:
+
       ```
       $ cwl-runner expression.cwl empty.yml
       [job 140000594593168] /home/example$ echo -A 2 -B baz -C 10 9 8 7 6 5 4 3 2 1
@@ -796,7 +806,7 @@
       class: Workflow
       ```
 
-      The 'cwlVersion` field indicates the version of the CWL spec used by the
+      The `cwlVersion` field indicates the version of the CWL spec used by the
       document.  The `class` field indicates this document describes a workflow.
 
 
@@ -820,7 +830,7 @@
 
       The `outputs` section describes the outputs of the workflow.  This is a
       list of output parameters where each parameter consists of an identifier
-      and a data type.  The `source` connects the output parameter `classfile`
+      and a data type.  The `outputSource` connects the output parameter `classfile`
       of the `compile` step to the workflow output parameter `classout`.
 
       ```

v1.1.0-dev1/concepts.md

@@ -89,7 +89,7 @@ preprocessing steps described in the
 [Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html).
 An implementation may formally validate the structure of a CWL document using
 SALAD schemas located at
-https://github.com/common-workflow-language/common-workflow-language/tree/master/draft-4
+https://github.com/common-workflow-language/common-workflow-language/tree/master/v1.1.0-dev1
 
 ## Identifiers