Skip to content

Commit 6cf9db8

Browse files
author
Peter Amstutz
committed
Add workflow example to user guide. Add missing files.
1 parent d34d2ac commit 6cf9db8

File tree

10 files changed

+257
-12
lines changed

10 files changed

+257
-12
lines changed

draft-3/UserGuide.yml

Lines changed: 181 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
doc:
44
- $include: userguide-intro.md
55

6+
- |
7+
# Wrapping Command Line Tools
8+
69
- |
710
## First example
811
@@ -11,9 +14,9 @@
1114
output. CWL documents are written in [JSON](http://json.org) or
1215
[YAML](http://yaml.org), or a mix of the two.
1316
14-
*1st.cwl*
17+
*1st-tool.cwl*
1518
```
16-
- $include: examples/1st.cwl
19+
- $include: examples/1st-tool.cwl
1720
- |
1821
```
1922
@@ -29,7 +32,7 @@
2932
command line:
3033
3134
```
32-
$ cwl-runner 1st.cwl echo-job.yml
35+
$ cwl-runner 1st-tool.cwl echo-job.yml
3336
[job 140199012414352] $ echo 'Hello world!'
3437
Hello world!
3538
Final process status is success
@@ -308,7 +311,7 @@
308311
Invoke `cwl-runner` with the tool wrapper and the input object on the
309312
command line:
310313
```
311-
$ cwl-runner example-tar-param.cwl example-tar-param-job.yml
314+
$ cwl-runner tar-param.cwl tar-param-job.yml
312315
[job 139868145165200] $ tar xf /home/example/hello.tar goodbye.txt
313316
Final process status is success
314317
{
@@ -475,8 +478,7 @@
475478
476479
```
477480
$ cwl-runner array-inputs.cwl array-inputs-job.yml
478-
/home/peter/work/cwl-venv/bin/cwltool 1.0.20160325210917
479-
[job 140334923640912] /home/peter/work/common-workflow-language/draft-3/examples$ echo -A one two three -B=four -B=five -B=six -C=seven,eight,nine
481+
[job 140334923640912] /home/example$ echo -A one two three -B=four -B=five -B=six -C=seven,eight,nine
480482
-A one two three -B=four -B=five -B=six -C=seven,eight,nine
481483
Final process status is success
482484
{}
@@ -492,6 +494,48 @@
492494
You can specify arrays of arrays, arrays of records, and other complex
493495
types.
494496
497+
- |
498+
## Array outputs
499+
500+
You can also capture multiple output files into an array of files using `glob`.
501+
502+
*array-outputs.cwl*
503+
```
504+
- $include: examples/array-outputs.cwl
505+
- |
506+
```
507+
508+
*array-outpust-job.yml*
509+
```
510+
- $include: examples/array-outputs-job.yml
511+
- |
512+
```
513+
514+
Now invoke `cwl-runner` providing the tool wrapper and the input object
515+
on the command line:
516+
517+
```
518+
$ cwl-runner array-outputs.cwl array-outputs-job.yml
519+
[job 140190876078160] /home/example$ touch foo.txt bar.dat baz.txt
520+
Final process status is success
521+
{
522+
"output": [
523+
{
524+
"size": 0,
525+
"path": "/home/peter/work/common-workflow-language/draft-3/examples/foo.txt",
526+
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
527+
"class": "File"
528+
},
529+
{
530+
"size": 0,
531+
"path": "/home/peter/work/common-workflow-language/draft-3/examples/baz.txt",
532+
"checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
533+
"class": "File"
534+
}
535+
]
536+
}
537+
```
538+
495539
- |
496540
## Record inputs, dependent and mutually exclusive parameters
497541
@@ -672,14 +716,143 @@
672716
673717
```
674718
$ cwl-runner linkfile.cwl arguments-job.yml
675-
[job 139928309171664] /home/example$ docker run -i --volume=/home/peter/work/common-workflow-language/draft-3/examples/Hello.java:/var/lib/cwl/job557617295_examples/Hello.java:ro --volume=/home/peter/work/common-workflow-language/draft-3/examples:/var/spool/cwl:rw --volume=/tmp/tmpmNbApw:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac Hello.java
719+
[job 139928309171664] /home/example$ docker run -i --volume=/home/example/Hello.java:/var/lib/cwl/job557617295_examples/Hello.java:ro --volume=/home/example:/var/spool/cwl:rw --volume=/tmp/tmpmNbApw:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac Hello.java
676720
Final process status is success
677721
{
678722
"classfile": {
679723
"size": 416,
680-
"path": "/home/peter/work/common-workflow-language/draft-3/examples/Hello.class",
724+
"path": "/home/example/Hello.class",
681725
"checksum": "sha1$2f7ac33c1f3aac3f1fec7b936b6562422c85b38a",
682726
"class": "File"
683727
}
684728
}
685729
```
730+
731+
- |
732+
# Writing Workflows
733+
734+
## First workflow
735+
736+
This workflow extracts a java source file from a tar file and then
737+
compiles it.
738+
739+
*1st-workflow.cwl*
740+
```
741+
- $include: examples/1st-workflow.cwl
742+
- |
743+
```
744+
745+
Use a JSON object in a separate file to describe the input of a run:
746+
747+
*1st-workflow-job.yml*
748+
```
749+
- $include: examples/1st-workflow-job.yml
750+
- |
751+
```
752+
753+
Now invoke `cwl-runner` with the tool wrapper and the input object on the
754+
command line:
755+
756+
```
757+
$ cwl-runner 1st-workflow.cwl 1st-workflow-job.yml
758+
[job untar] /tmp/tmp94qFiM$ tar xf /home/example/hello.tar Hello.java
759+
[step untar] completion status is success
760+
[job compile] /tmp/tmpu1iaKL$ docker run -i --volume=/tmp/tmp94qFiM/Hello.java:/var/lib/cwl/job301600808_tmp94qFiM/Hello.java:ro --volume=/tmp/tmpu1iaKL:/var/spool/cwl:rw --volume=/tmp/tmpfZnNdR:/tmp:rw --workdir=/var/spool/cwl --read-only=true --net=none --user=1001 --rm --env=TMPDIR=/tmp java:7 javac -d /var/spool/cwl /var/lib/cwl/job301600808_tmp94qFiM/Hello.java
761+
[step compile] completion status is success
762+
[workflow 1st-workflow.cwl] outdir is /home/example
763+
Final process status is success
764+
{
765+
"classout": {
766+
"path": "/home/example/Hello.class",
767+
"checksum": "sha1$2f7ac33c1f3aac3f1fec7b936b6562422c85b38a",
768+
"class": "File",
769+
"size": 416
770+
}
771+
}
772+
```
773+
774+
What's going on here? Let's break it down:
775+
776+
```
777+
cwlVersion: cwl:draft-3
778+
class: Workflow
779+
```
780+
781+
The 'cwlVersion` field indicates the version of the CWL spec used by the
782+
document. The `class` field indicates this document describes a workflow.
783+
784+
785+
```
786+
inputs:
787+
- id: inp
788+
type: File
789+
- id: ex
790+
type: string
791+
```
792+
793+
The `inputs` section describes the inputs of the workflow. This is a
794+
list of input parameters where each parameter consists of an identifier
795+
and a data type. These parameters can be used as sources for input to
796+
specific workflows steps.
797+
798+
```
799+
outputs:
800+
- id: classout
801+
type: File
802+
source: "#compile/classfile"
803+
```
804+
805+
The `outputs` section describes the outputs of the workflow. This is a
806+
list of output parameters where each parameter consists of an identifier
807+
and a data type. The `source` connects the output parameter `classfile`
808+
of the `compile` step to the workflow output parameter `classout`.
809+
810+
```
811+
steps:
812+
- id: untar
813+
run: tar-param.cwl
814+
inputs:
815+
- id: tarfile
816+
source: "#inp"
817+
- id: extractfile
818+
source: "#ex"
819+
outputs:
820+
- id: example_out
821+
```
822+
823+
The `steps` section describes the actual steps of the workflow. In this
824+
example, the first step extracts a file from a tar file, and the second
825+
step compiles the file from the first step using the java compiler.
826+
Workflow steps are not necessarily run in the order they are listed,
827+
instead the order is determined by the dependencies between steps (using
828+
`source`). In addition, workflow steps which do not depend on one
829+
another may run in parallel.
830+
831+
The first step, `untar` runs `tar-param.cwl` (described previously in
832+
[Parameter references](#Parameter_references)). This tool has two input
833+
parameters, `tarfile` and `extractfile` and one output parameter
834+
`example_out`.
835+
836+
The `inputs` section of the workflow step connects these two input
837+
parameters to the inputs of the workflow, `#inp` and `#ex` using
838+
`source`. This means that when the workflow step is executed, the values
839+
assigned to `#inp` and `#ex` will be used for the parameters `tarfile`
840+
and `extractfile` in order to run the tool.
841+
842+
The `outputs` section of the workflow step lists the output parameters
843+
that are expected from the tool.
844+
845+
```
846+
- id: compile
847+
run: arguments.cwl
848+
inputs:
849+
- id: src
850+
source: "#untar/example_out"
851+
outputs:
852+
- id: classfile
853+
```
854+
855+
The second step `compile` depends on the results from the first step by
856+
connecting the input parameter `src` to the output parameter of `untar`
857+
using `#untar/example_out`. The output of this step `classfile` is
858+
connected to the `outputs` section for the Workflow, described above.
File renamed without changes.

draft-3/examples/1st-workflow-job.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
inp:
2+
class: File
3+
path: hello.tar
4+
ex: Hello.java

draft-3/examples/1st-workflow.cwl

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
cwlVersion: cwl:draft-3
2+
class: Workflow
3+
inputs:
4+
- id: inp
5+
type: File
6+
- id: ex
7+
type: string
8+
9+
outputs:
10+
- id: classout
11+
type: File
12+
source: "#compile/classfile"
13+
14+
steps:
15+
- id: untar
16+
run: tar-param.cwl
17+
inputs:
18+
- id: tarfile
19+
source: "#inp"
20+
- id: extractfile
21+
source: "#ex"
22+
outputs:
23+
- id: example_out
24+
25+
- id: compile
26+
run: arguments.cwl
27+
inputs:
28+
- id: src
29+
source: "#untar/example_out"
30+
outputs:
31+
- id: classfile
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
touchfiles:
2+
- foo.txt
3+
- bar.dat
4+
- baz.txt

draft-3/examples/array-outputs.cwl

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
cwlVersion: cwl:draft-3
2+
class: CommandLineTool
3+
baseCommand: touch
4+
inputs:
5+
- id: touchfiles
6+
type:
7+
type: array
8+
items: string
9+
inputBinding:
10+
position: 1
11+
outputs:
12+
- id: output
13+
type:
14+
type: array
15+
items: File
16+
outputBinding:
17+
glob: "*.txt"

draft-3/examples/hello.tar

10 KB
Binary file not shown.

draft-3/userguide-intro.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ This guide will introduce you to writing tool wrappers and workflows using the
66
Common Workflow Language (CWL). This guide describes the current stable
77
specification, draft 3.
88

9+
Note: This document is a work in progress. Not all features are covered, yet.
10+
911
<!--ToC-->
1012

1113
# Introduction
@@ -24,5 +26,3 @@ technologies such as Docker, be used with CWL implementations from different
2426
vendors, and is well suited for describing large-scale workflows in cluster,
2527
cloud and high performance computing environments where tasks are scheduled in
2628
parallel across many nodes.
27-
28-
# Wrapping Command Line Tools

site/cwlsite.cwl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env cwl-runner
2-
cwlVersion: "cwl:draft-3.dev4"
2+
cwlVersion: cwl:draft-3
33

44
class: Workflow
55
inputs:

site/userguide-input.json

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"class": "File"
88
},
99
{
10-
"path": "../draft-3/examples/1st.cwl",
10+
"path": "../draft-3/examples/1st-tool.cwl",
1111
"class": "File"
1212
},
1313
{
@@ -70,6 +70,14 @@
7070
"path": "../draft-3/examples/array-inputs-job.yml",
7171
"class": "File"
7272
},
73+
{
74+
"path": "../draft-3/examples/array-outputs.cwl",
75+
"class": "File"
76+
},
77+
{
78+
"path": "../draft-3/examples/array-outputs-job.yml",
79+
"class": "File"
80+
},
7381
{
7482
"path": "../draft-3/examples/record.cwl",
7583
"class": "File"
@@ -113,6 +121,14 @@
113121
{
114122
"path": "../draft-3/examples/arguments-job.yml",
115123
"class": "File"
124+
},
125+
{
126+
"path": "../draft-3/examples/1st-workflow.cwl",
127+
"class": "File"
128+
},
129+
{
130+
"path": "../draft-3/examples/1st-workflow-job.yml",
131+
"class": "File"
116132
}
117133
]
118134
}

0 commit comments

Comments
 (0)