Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 17 16:28
    mr-c commented #868
  • Sep 17 13:42

    mr-c on 0.5.0

    (compare)

  • Sep 17 13:31

    mr-c on release_0.5

    (compare)

  • Sep 17 13:29
    mr-c opened #28
  • Sep 17 13:29

    mr-c on requires-io-master

    [requires.io] dependency update (compare)

  • Sep 17 13:29

    mr-c on requires-io-master

    [requires.io] dependency update (compare)

  • Sep 17 13:29

    mr-c on requires-io-master

    [requires.io] dependency update (compare)

  • Sep 17 13:28

    mr-c on requires-io-master

    (compare)

  • Sep 17 13:28

    mr-c on master

    add release-test format Merge pull request #27 from com… (compare)

  • Sep 17 13:28
    mr-c closed #27
  • Sep 17 09:11
    EricBoix closed #879
  • Sep 17 09:11
    EricBoix commented #879
  • Sep 17 09:10
    EricBoix edited #879
  • Sep 17 08:43
    EricBoix opened #1198
  • Sep 16 19:22
    mr-c synchronize #27
  • Sep 16 19:22

    mr-c on release_0.5

    format (compare)

  • Sep 16 18:43
    mr-c opened #27
  • Sep 16 18:43

    mr-c on release_0.5

    add release-test (compare)

  • Sep 16 18:33

    mr-c on release_0.5

    add release-test (compare)

  • Sep 16 18:32

    mr-c on release_0.5

    add release-test (compare)

Kaushik Ghose
@kaushik-work
@mr-c yes, sure thanks!
Michael Franklin
@illusional
Is it possible to import CWLTool in Python, and call a function to run a workflow? Currently I'm calling it via subprocess and sort of parsing stderr/stdout to figure out what's happening. But if there's a better way I'd love to do that.
Michael Franklin
@illusional
^^ I need to get the progress of the workflow too
pvanheus
@pvanheus
is there a way to return a array of arrays of File? I would like to turn an array of pairs of Files.
pvanheus
@pvanheus
ah:
        type: 
            type: array
            items: 
                type: array
                items: File
Kaushik Ghose
@kaushik-work
I wonder if this should also work
type: array
items: File[]
pvanheus
@pvanheus
@kaushik-work doesn't seem to: Fielditemsreferences unknown identifierFile[]``
Kaushik Ghose
@kaushik-work
@pvanheus ah, good to know that they syntactic sugar doesn't work when inside. It should be simple to resolve this, however.
Michael R. Crusoe
@mr-c
@tetron While reviewing the PR to add stderr to CWLType I came across this bug preventing us from building the website :-(
common-workflow-language/schema_salad#281
Peter Amstutz
@tetron
okay, I'll look at it in a bit
Michael R. Crusoe
@mr-c
Thanks!
pvanheus
@pvanheus
@kaushik-work well it works for now - I've got a tool that takes a dir full of FASTQ files and makes a list of lists of files and sample names... https://github.com/pvanheus/kraken2_workflow/blob/master/fastq_files_from_directory.cwl
might be useful. not quite sure where to put it. I'm using it as part of a kraken2 workflow I'm writing: https://github.com/pvanheus/kraken2_workflow/blob/master/kraken2_workflow.cwl
now to try and actually execute that with Cromwell or Toil or something :)
pvanheus
@pvanheus
I might have missed it in the best practices document - is there a preferred order for the sections? specifically should requirements go above inputs/outputs or below?
Kaushik Ghose
@kaushik-work
We've discussed this. There may be an issue, but I can't find it
Michael R. Crusoe
@mr-c
@pvanheus I've been using this order, but that isn't gospel: https://github.com/common-workflow-language/cwl-upgrader/blob/master/cwlupgrader/main.py#L390
pvanheus
@pvanheus
ok, I'll use it for now. requirements at the top. and $namespaces ? that isn't mentioned there
Michael R. Crusoe
@mr-c
At the bottom, along with $schema if you ask me
pvanheus
@pvanheus
yes it feels right...
pvanheus
@pvanheus
I'm trying to understand $import / $include - am I correct in understanding these can only be used to include a complete object (not, e.g. a list) ?
Kaushik Ghose
@kaushik-work
It can be a string (e.g in the case of JS libraries). In case of typedefs I've only ever seen dicts
The docs allow a more broad usage but in practice, it seems more restricted. common-workflow-language/common-workflow-language#879
pvanheus
@pvanheus
I read through the Schema Salad docs yesterday and still can't understand where and how $import can get used.
Michael R. Crusoe
@mr-c
Yeah, it isn't as usable as was hoped for
Michael R. Crusoe
@mr-c
Whoops, I didn't send out an agenda for today's meeting
Peter Amstutz
@tetron
this week's video chat is on! https://meet.jit.si/cwl
pvanheus
@pvanheus

I think I'm starting to understand the $import "'$include' must be the only field in ordereddict(...)" error - the CWL:

inputs:
  option1:
    $import: something.yml

is equivalent to:

inputs:
  - id: option1
    $import: something.yml

and thus the error.

and the Schema Salad specification says "It is an error if there are additional fields in the $import object, such additional fields must be ignored." (https://www.commonwl.org/v1.0/SchemaSalad.html#Import)
pvanheus
@pvanheus
so effectively $import can only replace a dict (object) with a dict (object). however the $mixin semantics seem more flexible

so...

inputs:
  option1:
    $mixin: something.yml

where something.yml is:

type:
  - type record:
    fields:
           ....
  - type: record
    fields:
         ....
pvanheus
@pvanheus
$mixin doesn't work with lists though - because the object (doc over here https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/ref_resolver.py#L728) is then a CommentedSeq which does not have a .update() method.
Anyway all of this stuff is venturing into the world of Schema Salad - which means you need to learn that language and its rules in top of CWL.
Michael R. Crusoe
@mr-c
Yeah, Schema Salad has always been part of the CWL spec. Anyone with a non-cwltool based implementation knows that too well and very few non-implementing users know about it as you now do
pvanheus
@pvanheus
Its a thing about workflow languages... each one is a gateway to a multitude of languages - Galaxy: Bash and XML, snakemake: Python, nextflow: Groovy, CWL: YAML, JavaScript (well ECMAScript 5) and Schema Salad.
How do you feel about $mixin ? I'm using it because I'm trying to extract "types" into "YAML macros" to satisfy the Dont Repeat Yourself principle, but I also feel I'm going too deep into "magic".
Peter Amstutz
@tetron
@pvanheus $mixin was never properly supported in the spec and conformance tests, I took it out in 1.1
pvanheus
@pvanheus
LOL!
Peter Amstutz
@tetron
@pvanheus sorry if you spent much time on it
pvanheus
@pvanheus
Ok so definitely not something to use. $import has the problem I highlighted earlier though...
Peter Amstutz
@tetron
so actually if you $import a list into a list, it expands into the parent list
[a, $import: b, c] where b = [b1, b2] turns into [a, b1, b2, c]
pvanheus
@pvanheus

ok... so you can do:

inputs:
  option1:
    type:
      $import: list-of-types.yml

but not

inputs:
  option1:
    $import: type-specification.yml
because the 2nd throws the '$import' must be the only field in ordereddict error
Peter Amstutz
@tetron
I think that's a schema salad bug
pvanheus
@pvanheus
well strictly it is speaking the truth, I think - option1 is another way of saying - id: option1 and then you have an object like (in JSON):
{ 
  "id": "option1",
  "$import": "type-specification.yml"
}
Peter Amstutz
@tetron
@pvanheus yes, you've got it. those features don't interact very well.
pvanheus
@pvanheus
CWL-as-I-write-it is compact, making use of "syntactic sugar" and it works till... it doesn't. Which is why I'd prefer to not hand-write CWL, but right now that seems to be the only option.