Merge pull request #244 from common-workflow-language/eliminate-dirent

Eliminate dirent

Author: mr-c

draft-4/CommandLineTool.yml

@@ -630,7 +630,7 @@ $graph:
         IRI such as an [RRID](http://www.identifiers.org/rrid/SCR_001156).
         Example: `http://identifiers.org/rrid/RRID:SCR_001156`
 
-- name: DirentExt
+- name: Dirent
   type: record
   doc: |
     Define a file or subdirectory that must be placed in the designated output
@@ -639,11 +639,12 @@ $graph:
     template.
   fields:
     - name: entryname
-      type: [string, Expression]
+      type: ["null", string, Expression]
       jsonldPredicate:
         _id: cwl:entryname
       doc: |
         The name of the file or subdirectory to create in the output directory.
+        If `entry` is a File or Directory, this overrides `basename`.  Optional.
     - name: entry
       type: [string, Expression]
       jsonldPredicate:
@@ -672,6 +673,7 @@ $graph:
         the original file or directory.  Default false (files and directories
         read-only by default).
 
+
 - name: InitialWorkDirRequirement
   type: record
   extends: ProcessRequirement
@@ -683,19 +685,18 @@ $graph:
     - name: listing
       type:
         - type: array
-          items: [File, DirentExt]
+          items: [File, Directory, Dirent, string, Expression]
         - string
         - Expression
       jsonldPredicate:
         _id: "cwl:listing"
-        mapSubject: entryname
-        mapPredicate: entry
       doc: |
         The list of files or subdirectories that must be placed in the
         designated output directory prior to executing the command line tool.
 
         May be an expression.  If so, the expression return value must validate
-        as `{type: array, items: [File, Dirent]}`.
+        as `{type: array, items: [File, Directory]}`.
+
 
 - name: EnvVarRequirement
   type: record

draft-4/Process.yml

@@ -65,7 +65,7 @@ $graph:
         _type: "@vocab"
       doc: Must be `File` to indicate this object describes a file.
     - name: location
-      type: string
+      type: string?
       doc: |
         A URI that identifies the file resource.  This may be a relative
         reference, in which case it must be resolved using the base URI of the
@@ -75,6 +75,10 @@ $graph:
         remote resource (due to unsupported protocol, access denied, or other
         issue) it must signal an error.
 
+        If the `location' field is not provided, the `contents` field must be
+        provided.  The implementation must assign a unique identifier for
+        the `location` field.
+
         If the `path` field is provided but the `location` field is not, an
         implementation may assign the value of the `path` field to `location`,
         then follow the rules above.
@@ -84,24 +88,34 @@ $graph:
     - name: path
       type: string?
       doc: |
-        The local path where the File is made available prior to executing a
-        CommandLineTool.  This must be set by the implementation.  This field
+        The local host path where the File is available when a CommandLineTool is
+        executed.  This field must be set by the implementation.  The final
+        path component must match the value of `basename`.  This field
         must not be used in any other context.  The command line tool being
         executed must be able to to access the file at `path` using the POSIX
         `open(2)` syscall.
+
+        As a special case, if the `path` field is provided but the `location`
+        field is not, an implementation may assign the value of the `path`
+        field to `location`, and remove the `path` field.
       jsonldPredicate:
         "_id": "cwl:path"
         "_type": "@id"
     - name: basename
       type: string?
       doc: |
-        The base name of the file, that is, the path component following the
-        final slash in the path.
-
-        The implementation must set this field based on the value of `path`
-        prior to evaluating parameter references or expressions in a
-        CommandLineTool document.  This field must not be used in any other
-        context.
+        The base name of the file, that is, the name of the file without any
+        leading directory path.  The base name must not contain a slash `/`.
+
+        If not provided, the implementation must set this field based on the
+        `location` field by taking the final path component after parsing
+        `location` as an IRI.  If `basename` is provided, it is not required to
+        match the value from `location`.
+
+        When this file is made available to a CommandLineTool, it must be named
+        with `basename`, i.e. the final component of the `path` field must match
+        `basename`.
+      jsonldPredicate: "cwl:basename"
     - name: dirname
       type: string?
       doc: |
@@ -121,10 +135,8 @@ $graph:
         period.  Leading periods on the basename are ignored; a basename of
         `.cshrc` will have a nameroot of `.cshrc`.
 
-        The implementation must set this field based on the value of `path`
-        prior to evaluating parameter references or expressions in a
-        CommandLineTool document.  This field must not be used in any other
-        context.
+        The implementation must set this field automatically based on the value
+        of `basename` prior to evaluating parameter references or expressions.
     - name: nameext
       type: string?
       doc: |
@@ -133,10 +145,8 @@ $graph:
         period.  Leading periods on the basename are ignored; a basename of
         `.cshrc` will have an empty `nameext`.
 
-        The implementation must set this field based on the value of `path`
-        prior to evaluating parameter references or expressions in a
-        CommandLineTool document.  This field must not be used in any other
-        context.
+        The implementation must set this field automatically based on the value
+        of `basename` prior to evaluating parameter references or expressions.
     - name: checksum
       type: ["null", string]
       doc: |
@@ -149,7 +159,7 @@ $graph:
       type:
         - "null"
         - type: array
-          items: [File, Dirent]
+          items: [File, Directory]
       jsonldPredicate: "cwl:secondaryFiles"
       doc: |
         A list of additional files that are associated with the primary file
@@ -179,26 +189,26 @@ $graph:
         File format ontologies may be provided in the "$schema" metadata at the
         root of the document.  If no ontologies are specified in `$schema`, the
         runtime may perform exact file format matches.
+    - name: contents
+      type: string?
+      doc: |
+        File contents literal.  Maximum of 64 KiB.
 
+        If neither `location` nor `path` is provided, `contents` must be
+        non-null.  The implementation must assign a unique identifier for the
+        `location` field.  When the file is staged as input to CommandLineTool,
+        the value of `contents` must be written to a file.
+
+        If `loadContents` of `inputBinding` or `outputBinding` is true and
+        `location` is valid, the implementation must read up to the first 64
+        KiB of text from the file and place it in the "contents" field.
 
-- name: Dirent
-  type: record
-  fields:
-    - name: entryname
-      type: string
-      jsonldPredicate:
-        "_id": cwl:entryname
-    - name: entry
-      type: [File, Directory]
-      jsonldPredicate:
-        "_id": cwl:entry
 
 - name: Directory
   type: record
   docParent: "#CWLType"
   doc: |
-    Represents a directory to present to a command line tool. This could be a virtual
-    directory, made of files assembled from multiple locations.
+    Represents a directory to present to a command line tool.
   fields:
     - name: class
       type:
@@ -211,7 +221,7 @@ $graph:
         _type: "@vocab"
       doc: Must be `Directory` to indicate this object describes a Directory.
     - name: location
-      type: string
+      type: string?
       doc: |
         A URI that identifies the directory resource.  This may be a relative
         reference, in which case it must be resolved using the base URI of the
@@ -222,6 +232,10 @@ $graph:
         unsupported protocol, access denied, or other issue) it must signal an
         error.
 
+        If the `location' field is not provided, the `listing` field must be
+        provided.  The implementation must assign a unique identifier for
+        the `location` field.
+
         If the `path` field is provided but the `location` field is not, an
         implementation may assign the value of the `path` field to `location`,
         then follow the rules above.
@@ -239,16 +253,36 @@ $graph:
       jsonldPredicate:
         _id: "cwl:path"
         _type: "@id"
+    - name: basename
+      type: string?
+      doc: |
+        The base name of the directory, that is, the name of the file without any
+        leading directory path.  The base name must not contain a slash `/`.
+
+        If not provided, the implementation must set this field based on the
+        `location` field by taking the final path component after parsing
+        `location` as an IRI.  If `basename` is provided, it is not required to
+        match the value from `location`.
+
+        When this file is made available to a CommandLineTool, it must be named
+        with `basename`, i.e. the final component of the `path` field must match
+        `basename`.
+      jsonldPredicate: "cwl:basename"
     - name: listing
       type:
         - "null"
         - type: array
-          items: [File, Dirent]
-      doc: List of files or subdirectories contained in this directory
+          items: [File, Directory]
+      doc: |
+        List of files or subdirectories contained in this directory.  The name
+        of each file or subdirectory is determined by the `basename` field of
+        each `File` or `Directory` object.  It is an error if a `File` shares a
+        `basename` with any other entry in `listing`.  If two or more
+        `Directory` object share the same `basename`, this must be treated as
+        equivalent to a single subdirectory with the listings recursively
+        merged.
       jsonldPredicate:
         _id: "cwl:listing"
-        mapSubject: entryname
-        mapPredicate: entry
 
 - name: SchemaBase
   type: record

draft-4/conformance_test_draft-4.yaml

@@ -732,18 +732,12 @@
         "class": "Directory",
         "listing": [
             {
-                "entryname": "hello.txt",
-                "entry": {
-                    "class": "File",
-                    "location": "hello.txt"
-                }
+              "class": "File",
+              "location": "hello.txt"
             },
             {
-                "entryname": "goodbye.txt",
-                "entry": {
-                    "class": "File",
-                    "location": "goodbye.txt"
-                }
+              "class": "File",
+              "location": "goodbye.txt"
             }
         ],
     }
@@ -785,3 +779,13 @@
   }
   tool: draft-4/stagefile.cwl
   doc: Test writable staged files.
+
+- job: draft-4/file-literal.yml
+  output:
+    output_file:
+      class: File
+      checksum: sha1$d0e04ff6c413c7d57f9a0ca0a33cd3ab52e2dd9c
+      location: output.txt
+      size: 18
+  tool: draft-4/cat3-tool.cwl
+  doc: Test file literal as input

draft-4/draft-4/dir4-job.yml

@@ -4,7 +4,6 @@ inf:
   secondaryFiles:
     - class: File
       location: index.py
-    - entryname: xtestdir
-      entry:
-        class: Directory
-        location: testdir
+    - class: Directory
+      basename: xtestdir
+      location: testdir

draft-4/draft-4/file-literal.yml

@@ -0,0 +1,3 @@
+file1:
+  class: File
+  contents: "Hello file literal"

draft-4/draft-4/search.cwl

@@ -9,7 +9,8 @@ $graph:
   requirements:
     - class: InitialWorkDirRequirement
       listing:
-        input.txt: $(inputs.file)
+        - entryname: input.txt
+          entry: $(inputs.file)
     - class: InlineJavascriptRequirement
 
   inputs:

draft-4/draft-4/stagefile.cwl

@@ -3,8 +3,7 @@ cwlVersion: draft-4.dev3
 requirements:
   InitialWorkDirRequirement:
     listing:
-      - entryname: $(inputs.infile.basename)
-        entry: $(inputs.infile)
+      - entry: $(inputs.infile)
         writable: true
 inputs:
   infile: File

draft-4/draft-4/template-tool.cwl

@@ -10,8 +10,8 @@ requirements:
       - "var t = function(s) { return _.template(s)({'inputs': inputs}); };"
   - class: InitialWorkDirRequirement
     listing:
-      foo.txt: >
-          $(t("The file is <%= inputs.file1.path.split('/').slice(-1)[0] %>\n"))
+      - entryname: foo.txt
+        entry: $(t("The file is <%= inputs.file1.path.split('/').slice(-1)[0] %>\n"))
 inputs:
   - id: file1
     type: File

draft-4/salad/schema_salad/jsonld_context.py

@@ -175,3 +175,52 @@ def salad_to_jsonld_context(j, schema_ctx):
         process_type(t, g, context, defaultBase, namespaces, defaultPrefix)
 
     return (context, g)
+
+def fix_jsonld_ids(obj, ids):
+    # type: (Union[Dict[unicode, Any], List[Dict[unicode, Any]]], List[unicode]) -> None
+    if isinstance(obj, dict):
+        for i in ids:
+            if i in obj:
+                obj["@id"] = obj[i]
+        for v in obj.values():
+            fix_jsonld_ids(v, ids)
+    if isinstance(obj, list):
+        for entry in obj:
+            fix_jsonld_ids(entry, ids)
+
+def makerdf(workflow, wf, ctx):
+    # type: (Union[str, unicode], Union[List[Dict[unicode, Any]], Dict[unicode, Any]], Loader.ContextType) -> Graph
+    prefixes = {}
+    idfields = []
+    for k,v in ctx.iteritems():
+        if isinstance(v, dict):
+            url = v["@id"]
+        else:
+            url = v
+        if url == "@id":
+            idfields.append(k)
+        doc_url, frg = urlparse.urldefrag(url)
+        if "/" in frg:
+            p, _ = frg.split("/")
+            prefixes[p] = u"%s#%s/" % (doc_url, p)
+
+    if isinstance(wf, list):
+        wf = {
+            "@context": ctx,
+            "@graph": wf
+        }
+    else:
+        wf["@context"] = ctx
+
+    fix_jsonld_ids(wf, idfields)
+
+    g = Graph().parse(data=json.dumps(wf), format='json-ld', location=workflow)
+
+    # Bug in json-ld loader causes @id fields to be added to the graph
+    for s,p,o in g.triples((None, URIRef("@id"), None)):
+        g.remove((s, p, o))
+
+    for k2,v2 in prefixes.iteritems():
+        g.namespace_manager.bind(k2, v2)
+
+    return g