Squashed 'cwltool/schemas/' changes from f027dbb..d03b190

d03b190 Introducing refScope and typeDSL into CWL schema
5e1e438 Merge commit '851b3c1302aa5b5d8a80f74c5732a617f62acefe' into scoped-ref-and-typedsl
851b3c1 Split minimal base types used in CWL into metaschema_base.yml
510a01f Merge commit '8823aaa11432aa042b3e307891e71ee13e11bfd7' into scoped-ref-and-typedsl
8823aaa Adjust ref scopes
3431e18 Use scopedRef and typeDSL in metaschema
1e6dab6 Using typeDSL in metaschema.
3055f70 Update bootstrap context.
31db28e Merge branch 'master' into type_dsl.  Add test for typeDSL.
116d9e4 Merge branch 'resolve_all_refactor'
8072bf5 "type" DSL wip
33ef215 Refactor resolve_all into smaller methods.
ad8a111 Merge pull request #28 from common-workflow-language/scoped_ref
5aadf27 fix types
4ef52be Improve documentation.  Add feature of skipping some inner scopes.
c6d4a7b Reworking refScope to be fixed instead of search based.
ae0ef4e Remove explicit calls to validate_links since it is folded into resolve_all().
2fa6bfb update types
e5d4730 New test for scoped ref that better reflects actual desired behavior.
efbc9c4 Add scopedRef to metaschema.  Tweak test.
9fcd67d Fold link validation into resolve_all() and resolve scoped identifiers.
f36f52f Merge pull request #219 from denis-yuen/patch-1
da10eec always be updating (mypy-lang)
e1e17ea upgrade Jenkins setuptools
563e5ef fix jenkins build
11b45b5 Merge pull request #224 from common-workflow-language/jenkins-link
b4d790d Backport fixes to draft-3
1ea9a5c add jenkins link
180a21e Corrections and test data
f0ac285 Merge branch 'master' of github.com:common-workflow-language/schema_salad
84738f1 better RDF schema error handling (#26)
3b8dc09 mypy 0.4 is out (#25)
5316da0 Add missing files for site generation.
31fbb76 Fix site generation workflows to specify draft-3
cff09a0 Add link to latest User Guide on front page
53b2ad7 Merge pull request #200 from common-workflow-language/draft-4
089fbd7 start documenting the draft-4 changes
8c5bda0 Fix mapSubject fields when $import or $include is provided.
5b969ca fix diff-cover
d798a7e split out mypy
f43fa45 Fix tests
2d0398e Bump version.
aeab7c3 Merge branch 'master' of github.com:common-workflow-language/schema_salad
e381f64 Return metaschema loader from load_schema.  Don't crash expanding identifier fields when value is not a string.
15dae9f enhance CI
fd2fcb3 more type checking (#21)
c51c723 enable rdfa schema format (#22)
ff192e8 turn off py3 support for now (#19)
0c5dba5 measure code coverage (#12)
2514a23 switch to ruamel.yaml, use C loader if available, only load safely (#20)

git-subtree-dir: cwltool/schemas
git-subtree-split: d03b190

Author: Peter Amstutz

README.md

@@ -14,6 +14,11 @@ CWL builds on technologies such as [JSON-LD](http://json-ld.org) and
 CWL is designed to express workflows for data-intensive science, such as
 Bioinformatics, Medical Imaging, Chemistry, Physics, and Astronomy.
 
+## CWL User Guide
+
+[User guide for the current stable specification (draft-3)](http://www.commonwl.org/draft-3/UserGuide.html),
+provides a gentle introduction to writing CWL command line tool and workflow descriptions.
+
 ## CWL Specification
 
 The current stable specification is [draft 3](http://www.commonwl.org/draft-3/):
@@ -55,6 +60,10 @@ Some of the software supporting running Common Workflow Language tools or workfl
 * [Apache Taverna](http://taverna.incubator.apache.org/),
   [Apache Taverna wiki page](https://github.com/common-workflow-language/common-workflow-language/wiki/Taverna)
 
+We continuously run the CWL conformance tests on several implementations:
+
+https://ci.commonwl.org
+
 ## Examples
 
 [Github repository of example tools and workflows.](https://github.com/common-workflow-language/workflows)

draft-3/UserGuide.yml

@@ -20,7 +20,7 @@
     - |
       ```
 
-      Use a JSON object in a separate file to describe the input of a run:
+      Use a YAML object in a separate file to describe the input of a run:
 
       *echo-job.yml*
       ```
@@ -106,10 +106,11 @@
       Notice that "example_file", as a `File` type, must be provided as an
       object with the fields `class: File` and `path`.
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a whale.txt and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
 
       ```
+      $ touch whale.txt
       $ cwl-runner inp.cwl inp-job.yml
       [job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
       -f -i42 --example-string hello --file=/home/example/whale.txt
@@ -218,9 +219,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a tar file for the example and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ touch hello.txt && tar -cvf hello.tar hello.txt
       $ cwl-runner tar.cwl tar-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar
       Final process status is success
@@ -308,9 +310,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Create your input files and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar
       $ cwl-runner tar-param.cwl tar-param-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
       Final process status is success
@@ -374,11 +377,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Provide a hello.js and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-docker.cwl example-docker-job.yml
+      $ echo "console.log(\"Hello World\");" > hello.js
+      $ cwl-runner docker.cwl docker-job.yml
       [job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
       Hello world!
       Final process status is success
@@ -417,11 +421,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Now create a sample Java file and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-arguments.cwl example-arguments-job.yml
+      $ echo "public class Hello {}" > Hello.java
+      $ cwl-runner arguments.cwl arguments-job.yml
       [job 140051188854928] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job710906416_example/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpdlQDWi:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job710906416_examples/Hello.java
       Final process status is success
       {
@@ -855,4 +860,4 @@
       The second step `compile` depends on the results from the first step by
       connecting the input parameter `src` to the output parameter of `untar`
       using `#untar/example_out`.  The output of this step `classfile` is
-      connected to the `outputs` section for the Workflow, described above.
+      connected to the `outputs` section for the Workflow, described above.

draft-4/CommandLineTool.yml

@@ -38,8 +38,12 @@ $graph:
     - |
       ## Introduction to draft 4
 
-      This specification represents the third milestone of the CWL group.
-      Since draft-3, this draft introduces the following major changes and additions:
+      This specification represents the fourth milestone of the CWL group.
+      Since draft-3, this draft introduces the following changes and additions:
+
+        * Preliminary support for initialWorkDir & the Directory type
+        * "id: name" is now just "name"; "class: Classname" is now just
+          "Classtname".
 
       ## Purpose
 

draft-4/Process.yml

@@ -10,7 +10,7 @@ $graph:
   type: documentation
   doc: {$include: concepts.md}
 
-- $import: "salad/schema_salad/metaschema/metaschema.yml"
+- $import: "salad/schema_salad/metaschema/metaschema_base.yml"
 
 - name: BaseTypesDoc
   type: documentation
@@ -226,6 +226,8 @@ $graph:
       jsonldPredicate:
         "_id": "sld:type"
         "_type": "@vocab"
+        refScope: 2
+        typeDSL: True
       doc: |
         Specify valid types of data that may be assigned to this parameter.
 

draft-4/UserGuide.yml

@@ -17,7 +17,7 @@
     - |
       ```
 
-      Use a JSON object in a separate file to describe the input of a run:
+      Use a YAML object in a separate file to describe the input of a run:
 
       *echo-job.yml*
       ```
@@ -103,10 +103,11 @@
       Notice that "example_file", as a `File` type, must be provided as an
       object with the fields `class: File` and `path`.
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a whale.txt and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
 
       ```
+      $ touch whale.txt
       $ cwl-runner inp.cwl inp-job.yml
       [job 140020149614160] /home/example$ echo -f -i42 --example-string hello --file=/home/example/whale.txt
       -f -i42 --example-string hello --file=/home/example/whale.txt
@@ -215,9 +216,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Next, create a tar file for the example and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ touch hello.txt && tar -cvf hello.tar hello.txt
       $ cwl-runner tar.cwl tar-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar
       Final process status is success
@@ -305,9 +307,10 @@
     - |
       ```
 
-      Invoke `cwl-runner` with the tool wrapper and the input object on the
+      Create your input files and invoke `cwl-runner` with the tool wrapper and the input object on the
       command line:
       ```
+      $ rm hello.tar || true && touch goodbye.txt && tar -cvf hello.tar
       $ cwl-runner example-tar-param.cwl example-tar-param-job.yml
       [job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
       Final process status is success
@@ -371,11 +374,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Provide a hello.js and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-docker.cwl example-docker-job.yml
+      $ echo "console.log(\"Hello World\");" > hello.js
+      $ cwl-runner docker.cwl docker-job.yml
       [job 140259721854416] /home/example$ docker run -i --volume=/home/example/hello.js:/var/lib/cwl/job369354770_examples/hello.js:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpDLs5hm:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp node:slim node /var/lib/cwl/job369354770_examples/hello.js
       Hello world!
       Final process status is success
@@ -414,11 +418,12 @@
     - |
       ```
 
-      Now invoke `cwl-runner` providing the tool wrapper and the input object
+      Now create a sample Java file and invoke `cwl-runner` providing the tool wrapper and the input object
       on the command line:
 
       ```
-      $ cwl-runner example-arguments.cwl example-arguments-job.yml
+      $ echo "public class Hello {}" > Hello.java
+      $ cwl-runner arguments.cwl arguments-job.yml
       [job 140051188854928] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job710906416_example/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpdlQDWi:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job710906416_examples/Hello.java
       Final process status is success
       {