Merge commit '8348c9d6b69bee117c325bf52081dc7bf3ed8502' into stderr

Author: mr-c

cwltool/schemas/README.md

@@ -14,6 +14,11 @@ CWL builds on technologies such as [JSON-LD](http://json-ld.org) and
 CWL is designed to express workflows for data-intensive science, such as
 Bioinformatics, Medical Imaging, Chemistry, Physics, and Astronomy.
 
+## CWL User Guide
+
+[User guide for the current stable specification (draft-3)](http://www.commonwl.org/draft-3/UserGuide.html),
+provides a gentle introduction to writing CWL command line tool and workflow descriptions.
+
 ## CWL Specification
 
 The current stable specification is [draft 3](http://www.commonwl.org/draft-3/):
@@ -55,6 +60,10 @@ Some of the software supporting running Common Workflow Language tools or workfl
 * [Apache Taverna](http://taverna.incubator.apache.org/),
   [Apache Taverna wiki page](https://github.com/common-workflow-language/common-workflow-language/wiki/Taverna)
 
+We continuously run the CWL conformance tests on several implementations:
+
+https://ci.commonwl.org
+
 ## Examples
 
 [Github repository of example tools and workflows.](https://github.com/common-workflow-language/workflows)

cwltool/schemas/draft-3/UserGuide.yml

@@ -20,7 +20,7 @@
     - |
       ```
 
-      Use a JSON object in a separate file to describe the input of a run:
+      Use a YAML object in a separate file to describe the input of a run:
 
       *echo-job.yml*
       ```
@@ -106,10 +106,11 @@
       Notice that "example_file", as a `File` type, must be provided as an
       object with the fields `class: File` and `path`.
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a whale.txt and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
 
       ```
+      $ touch whale.txt
       $ cwl-runner inp.cwl inp-job.yml
       [job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
       -f -i42 --example-string hello --file=/home/example/whale.txt
@@ -218,9 +219,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a tar file for the example and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ touch hello.txt && tar -cvf hello.tar hello.txt
       $ cwl-runner tar.cwl tar-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar
       Final process status is success
@@ -308,9 +310,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Create your input files and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar
       $ cwl-runner tar-param.cwl tar-param-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
       Final process status is success
@@ -374,11 +377,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Provide a hello.js and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-docker.cwl example-docker-job.yml
+      $ echo "console.log(\"Hello World\");" > hello.js
+      $ cwl-runner docker.cwl docker-job.yml
       [job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
       Hello world!
       Final process status is success
@@ -417,11 +421,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Now create a sample Java file and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-arguments.cwl example-arguments-job.yml
+      $ echo "public class Hello {}" > Hello.java
+      $ cwl-runner arguments.cwl arguments-job.yml
       [job 140051188854928] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job710906416_example/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpdlQDWi:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job710906416_examples/Hello.java
       Final process status is success
       {
@@ -855,4 +860,4 @@
       The second step `compile` depends on the results from the first step by
       connecting the input parameter `src` to the output parameter of `untar`
       using `#untar/example_out`.  The output of this step `classfile` is
-      connected to the `outputs` section for the Workflow, described above.
+      connected to the `outputs` section for the Workflow, described above.

cwltool/schemas/draft-3/cwl-runner.cwl

@@ -0,0 +1,78 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: "cwl:draft-3"
+
+description: |
+ Generic interface to run a Common Workflow Language tool or workflow from the
+ command line. To be implemented by each CWL compliant execution platform for
+ testing conformance to the standard and optionally for use by users.
+
+inputs:
+ - id: outdir
+   type: string
+   default: outdir
+   description: |
+    Output directory, defaults to the current directory
+   inputBinding:
+    prefix: "--outdir"
+
+ - id: quiet
+   type: boolean
+   description: no diagnostic output
+   inputBinding:
+    prefix: "--quiet"
+
+ - id: toolfile
+   type: [ "null", File ]
+   description: |
+    The tool or workflow description to run. Optional if the jobfile has a
+    `cwl:tool` field to indicate the tool or workflow description to run.
+   inputBinding:
+    position: 1
+
+ - id: jobfile
+   type: File
+   inputBinding:
+    position: 2
+
+ - id: conformance-test
+   type: boolean
+   inputBinding:
+    prefix: "--conformance-test"
+
+ - id: basedir
+   type: string
+   inputBinding:
+    prefix: "--basedir"
+
+ - id: no-container
+   type: boolean
+   description: |
+    Do not execute jobs in a Docker container, even when listed as a Requirement
+   inputBinding:
+    prefix: "--no-container"
+
+ - id: tmp-outdir-prefix
+   type: string
+   description: |
+    Path prefix for temporary directories. Useful for OS X so that boot2docker
+    writes to /Users
+   inputBinding:
+    prefix: "--tmp-outdir-prefix"
+
+ - id: tmpdir-prefix
+   type: string
+   description: |
+    Path prefix for temporary directories
+   inputBinding:
+    prefix: "--tmpdir-prefix"
+
+baseCommand: cwl-runner
+
+stdout: output-object.json
+
+outputs:
+ - id: output-object
+   type: File
+   outputBinding:
+     glob: output-object.json

cwltool/schemas/draft-4/CommandLineTool.yml

@@ -38,8 +38,12 @@ $graph:
     - |
       ## Introduction to draft 4
 
-      This specification represents the third milestone of the CWL group.
-      Since draft-3, this draft introduces the following major changes and additions:
+      This specification represents the fourth milestone of the CWL group.
+      Since draft-3, this draft introduces the following changes and additions:
+
+        * Preliminary support for initialWorkDir & the Directory type
+        * "id: name" is now just "name"; "class: Classname" is now just
+          "Classtname".
 
       ## Purpose
 
@@ -430,6 +434,18 @@ $graph:
       doc: |
         A path to a file whose contents must be piped into the command's
         standard input stream.
+    - name: stderr
+      type: ["null", string, "#Expression"]
+      doc: |
+        Capture the command's standard error stream to a file written to
+        the designated output directory.
+
+        If `stderr` is a string, it specifies the file name to use.
+
+        If `stderr` is an expression, the expression is evaluated and must
+        return a string with the file name to use to capture stderr.  If the
+        return value is not a string, or the resulting path contains illegal
+        characters (such as the path separator `/`) it is an error.
     - name: stdout
       type: ["null", string, "#Expression"]
       doc: |

cwltool/schemas/draft-4/UserGuide.yml

@@ -17,7 +17,7 @@
     - |
       ```
 
-      Use a JSON object in a separate file to describe the input of a run:
+      Use a YAML object in a separate file to describe the input of a run:
 
       *echo-job.yml*
       ```
@@ -103,10 +103,11 @@
       Notice that "example_file", as a `File` type, must be provided as an
       object with the fields `class: File` and `path`.
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a whale.txt and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
 
       ```
+      $ touch whale.txt
       $ cwl-runner inp.cwl inp-job.yml
       [job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
       -f -i42 --example-string hello --file=/home/example/whale.txt
@@ -215,9 +216,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a tar file for the example and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ touch hello.txt && tar -cvf hello.tar hello.txt
       $ cwl-runner tar.cwl tar-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar
       Final process status is success
@@ -305,9 +307,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Create your input files and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar
       $ cwl-runner example-tar-param.cwl example-tar-param-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
       Final process status is success
@@ -371,11 +374,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Provide a hello.js and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-docker.cwl example-docker-job.yml
+      $ echo "console.log(\"Hello World\");" > hello.js
+      $ cwl-runner docker.cwl docker-job.yml
       [job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
       Hello world!
       Final process status is success
@@ -414,11 +418,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Now create a sample Java file and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-arguments.cwl example-arguments-job.yml
+      $ echo "public class Hello {}" > Hello.java
+      $ cwl-runner arguments.cwl arguments-job.yml
       [job 140051188854928] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job710906416_example/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpdlQDWi:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job710906416_examples/Hello.java
       Final process status is success
       {

cwltool/schemas/draft-4/Workflow.yml

@@ -34,10 +34,10 @@ $graph:
 
       ## Introduction to draft 4
 
-      This specification represents the fourth milestone of the CWL group.  Since
-      draft-3, this draft introduces the following changes and additions:
+      This specification represents the fourth milestone of the CWL group.
+      Since draft-3, this draft introduces the following changes and additions:
 
-        * TBD
+        * The 'inputs' and 'outputs' fields have been renamed 'in' and 'out'.
 
       ## Purpose
 
@@ -469,4 +469,4 @@ $graph:
   extends: "#ProcessRequirement"
   doc: |
     Indicate that the workflow platform must support the `valueFrom` field
-    of [WorkflowStepInput](#WorkflowStepInput).
+    of [WorkflowStepInput](#WorkflowStepInput).

cwltool/schemas/draft-4/conformance_test_draft-4.yaml

@@ -44,6 +44,12 @@
   tool: draft-4/cat3-tool.cwl
   doc: Test command line with stdout redirection
 
+- args: [egrep]
+  stderr: error.txt
+  job:
+  tool: draft-4/egrep-stderr.cwl
+  doc: Test command line with stdout redirection
+
 - args: [cat]
   job: draft-4/cat-job.json
   stdin: draft-4/hello.txt