common-workflow-language/common-workflow-language

Kaushik Ghose

@kaushik-work

@mr-c yes, sure thanks!

Michael Franklin

@illusional

Is it possible to import CWLTool in Python, and call a function to run a workflow? Currently I'm calling it via subprocess and sort of parsing stderr/stdout to figure out what's happening. But if there's a better way I'd love to do that.

Michael Franklin

@illusional

^^ I need to get the progress of the workflow too

pvanheus

@pvanheus

is there a way to return a array of arrays of File? I would like to turn an array of pairs of Files.

pvanheus

@pvanheus

ah:

        type: 
            type: array
            items: 
                type: array
                items: File

Kaushik Ghose

@kaushik-work

I wonder if this should also work

type: array
items: File[]

pvanheus

@pvanheus

@kaushik-work doesn't seem to: Fielditemsreferences unknown identifierFile[]``

Kaushik Ghose

@kaushik-work

@pvanheus ah, good to know that they syntactic sugar doesn't work when inside. It should be simple to resolve this, however.

Michael R. Crusoe

@mr-c

@tetron While reviewing the PR to add stderr to CWLType I came across this bug preventing us from building the website :-(
common-workflow-language/schema_salad#281

Peter Amstutz

@tetron

okay, I'll look at it in a bit

Michael R. Crusoe

@mr-c

Thanks!

pvanheus

@pvanheus

@kaushik-work well it works for now - I've got a tool that takes a dir full of FASTQ files and makes a list of lists of files and sample names... https://github.com/pvanheus/kraken2_workflow/blob/master/fastq_files_from_directory.cwl

might be useful. not quite sure where to put it. I'm using it as part of a kraken2 workflow I'm writing: https://github.com/pvanheus/kraken2_workflow/blob/master/kraken2_workflow.cwl

now to try and actually execute that with Cromwell or Toil or something :)

pvanheus

@pvanheus

I might have missed it in the best practices document - is there a preferred order for the sections? specifically should requirements go above inputs/outputs or below?

Kaushik Ghose

@kaushik-work

We've discussed this. There may be an issue, but I can't find it

Michael R. Crusoe

@mr-c

@pvanheus I've been using this order, but that isn't gospel: https://github.com/common-workflow-language/cwl-upgrader/blob/master/cwlupgrader/main.py#L390

pvanheus

@pvanheus

ok, I'll use it for now. requirements at the top. and $namespaces ? that isn't mentioned there

Michael R. Crusoe

@mr-c

At the bottom, along with $schema if you ask me

pvanheus

@pvanheus

yes it feels right...

pvanheus

@pvanheus

I'm trying to understand $import / $include - am I correct in understanding these can only be used to include a complete object (not, e.g. a list) ?

Kaushik Ghose

@kaushik-work

It can be a string (e.g in the case of JS libraries). In case of typedefs I've only ever seen dicts

https://github.com/common-workflow-language/common-workflow-language/blob/f24c797ea017a467185b516ea4862c9c494c9d33/v1.0/v1.0/template-tool.cwl

The docs allow a more broad usage but in practice, it seems more restricted. common-workflow-language/common-workflow-language#879

pvanheus

@pvanheus

I read through the Schema Salad docs yesterday and still can't understand where and how $import can get used.

Michael R. Crusoe

@mr-c

Yeah, it isn't as usable as was hoped for

Michael R. Crusoe

@mr-c

Whoops, I didn't send out an agenda for today's meeting

Peter Amstutz

@tetron

this week's video chat is on! https://meet.jit.si/cwl

pvanheus

@pvanheus

I think I'm starting to understand the $import "'$include' must be the only field in ordereddict(...)" error - the CWL:

inputs:
  option1:
    $import: something.yml

is equivalent to:

inputs:
  - id: option1
    $import: something.yml

and thus the error.

and the Schema Salad specification says "It is an error if there are additional fields in the $import object, such additional fields must be ignored." (https://www.commonwl.org/v1.0/SchemaSalad.html#Import)

pvanheus

@pvanheus

the code handling this is: https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/ref_resolver.py#L618

pvanheus

@pvanheus

so effectively $import can only replace a dict (object) with a dict (object). however the $mixin semantics seem more flexible

so...

inputs:
  option1:
    $mixin: something.yml

where something.yml is:

type:
  - type record:
    fields:
           ....
  - type: record
    fields:
         ....

pvanheus

@pvanheus

$mixin doesn't work with lists though - because the object (doc over here https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/ref_resolver.py#L728) is then a CommentedSeq which does not have a .update() method.

Anyway all of this stuff is venturing into the world of Schema Salad - which means you need to learn that language and its rules in top of CWL.

Michael R. Crusoe

@mr-c

Yeah, Schema Salad has always been part of the CWL spec. Anyone with a non-cwltool based implementation knows that too well and very few non-implementing users know about it as you now do

pvanheus

@pvanheus

Its a thing about workflow languages... each one is a gateway to a multitude of languages - Galaxy: Bash and XML, snakemake: Python, nextflow: Groovy, CWL: YAML, JavaScript (well ECMAScript 5) and Schema Salad.

How do you feel about $mixin ? I'm using it because I'm trying to extract "types" into "YAML macros" to satisfy the Dont Repeat Yourself principle, but I also feel I'm going too deep into "magic".

Peter Amstutz

@tetron

@pvanheus $mixin was never properly supported in the spec and conformance tests, I took it out in 1.1

pvanheus

@pvanheus

LOL!

Peter Amstutz

@tetron

@pvanheus sorry if you spent much time on it

pvanheus

@pvanheus

Ok so definitely not something to use. $import has the problem I highlighted earlier though...

Peter Amstutz

@tetron

so actually if you $import a list into a list, it expands into the parent list

[a, $import: b, c] where b = [b1, b2] turns into [a, b1, b2, c]

pvanheus

@pvanheus

ok... so you can do:

inputs:
  option1:
    type:
      $import: list-of-types.yml

but not

inputs:
  option1:
    $import: type-specification.yml

because the 2nd throws the '$import' must be the only field in ordereddict error

Peter Amstutz

@tetron

I think that's a schema salad bug

pvanheus

@pvanheus

well strictly it is speaking the truth, I think - option1 is another way of saying - id: option1 and then you have an object like (in JSON):

{ 
  "id": "option1",
  "$import": "type-specification.yml"
}

Peter Amstutz

@tetron

@pvanheus yes, you've got it. those features don't interact very well.

pvanheus

@pvanheus

CWL-as-I-write-it is compact, making use of "syntactic sugar" and it works till... it doesn't. Which is why I'd prefer to not hand-write CWL, but right now that seems to be the only option.

Where communities thrive

People

Repo info

Activity