Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 06 07:55
    mr-c review_requested #1703
  • Aug 06 07:13
    kinow edited #1703
  • Aug 06 07:13
    kinow edited #1703
  • Aug 06 07:11
    kinow commented #1703
  • Aug 06 07:06
    kinow ready_for_review #1703
  • Aug 06 06:28
    codecov[bot] commented #1703
  • Aug 06 06:20
    codecov[bot] commented #1703
  • Aug 06 06:18
    kinow commented #1703
  • Aug 06 06:17
    codecov[bot] commented #1703
  • Aug 06 06:17
    codecov[bot] commented #1703
  • Aug 06 06:16
    codecov[bot] commented #1703
  • Aug 06 06:16
    codecov[bot] commented #1703
  • Aug 06 06:13
    codecov[bot] commented #1703
  • Aug 06 06:13
    codecov[bot] commented #1703
  • Aug 06 06:12
    codecov[bot] commented #1703
  • Aug 06 06:10
    codecov[bot] commented #1703
  • Aug 06 06:09
    codecov[bot] commented #1703
  • Aug 06 06:09
    codecov[bot] commented #1703
  • Aug 06 06:02
    codecov[bot] commented #1703
  • Aug 06 06:02
    codecov[bot] commented #1703
Adrian Zimmer
@zimmera:matrix.org
[m]
Im afraid i will be too busy writing my thesis. Just started with that this week πŸ˜•
1 reply
Can I think about it and tell you later this day or am I already late?
Michael Crusoe
@mr-c:matrix.org
[m]
Later is fine, yep!
Adrian Zimmer
@zimmera:matrix.org
[m]
Alright thanks! πŸ™‚
Michael Crusoe
@mr-c:matrix.org
[m]
Many thank Melissa Black for taking the @commonwl twitter account to new heights!
Adrian Zimmer
@zimmera:matrix.org
[m]
Michael Crusoe: I decided to hold a talk about cwl-ts-auto and most likely also how we use it in ARCs. πŸ˜€
What date is the deadline for uploading the actual videos?
2 replies
glassofwhiskey
@glassofwhiskey:matrix.org
[m]
Hi, sorry to go OT but can someone tell me what is the support level for streaming data in CWL? I mean, when step A writed a stream and step B reads from that stream
1 reply
Michael Crusoe
@mr-c:matrix.org
[m]

Arvados automatically streams all inputs and outputs from their datastore, Keep. toil-cwl-runner can stream from or two an object store, if that is configured.

Direct streaming between steps is permitted by engines, but difficult to setup

glassofwhiskey
@glassofwhiskey:matrix.org
[m]
Thank you for the detailed answer ^^
Michael Crusoe
@mr-c:matrix.org
[m]
glassofwhiskey: you are welcome! Here is how Toil implemented streaming to/from object stores: DataBiosphere/toil#3694
Michael Crusoe
@mr-c:matrix.org
[m]
In common-workflow-language/cwltool#1605 I'm extending --validate to work on packed CWL documents ($graph) that don't define a #main or main Process. Theoretically this could break things for people who want the error. Peter Amstutz and others, do we think this is safe to do so, or should I add another command line option (--validate-all) for this new behavior?
4 replies
Peter Amstutz
@tetron:matrix.org
[m]
sounds fine
Michael Crusoe
@mr-c:matrix.org
[m]
Thanks!
Adrian Zimmer
@zimmera:matrix.org
[m]
Just handed in my talk submission (better late than never πŸ˜„)
1 reply
Jasper Koehorst
@jjkoehorst_gitlab
was it already possible in cwl to have stdout to stdout and to a file? this is to have the stdout to a log file but to also see how the progress is going. or does cwl print it directly to the file without buffer so we can track some sort of progress there?
4 replies
or is this included in the provenance folder?
Kevin Hannon
@kannon92

I tried to find this in the code but I wasn't able too. How exactly does the name of the job get calculated?

I notice that toil doesn't change the name of the job based on the scatter index. This causes a minor problem in my real time logging. But I don't really know how to give a name of the scatter to cwltool. I think toil takes the name of the plugin.

Michael Crusoe
@mr-c:matrix.org
[m]

I tried to find this in the code but I wasn't able too. How exactly does the name of the job get calculated?

In cwltool?

Melissa Black
@melibleq:matrix.org
[m]
Hello everyone! Another CWL weekly meeting starts in 23 minutes.
Agenda & notes: https://w3id.org/cwl/meeting_minutes
Direct link to video: https://meet.jit.si/CompetentBranchesNominateWeakly
isubasinghe
@isubasinghe:matrix.org
[m]
Sorry Michael, I should check this chat more often but I have not implemented conditionals yet
I have implemented file support only recently
Tomoya Tanjo
@tom-tan

Hello!

I made three custom Github Actions for CWL!
I hope they make easier to test users' workflows and also make developers of workflow engines easier to run conformance tests and to publish the results on their website.

Currently they are my personal repositories but it would be nice if they are provided as CWL official repositories to make them sustainable.

Can I move these repositories to CWL organization?

7 replies
Adrian Zimmer
@zimmera:matrix.org
[m]
Does anyone know how I can tell cwltool/Docker to re-build a Dockerfile after I changed it? I updated a version number of a tool in my Dockerfile but cwltool still uses the old version
Adrian Zimmer
@zimmera:matrix.org
[m]
I think I fixed it...
Michael Crusoe
@mr-c:matrix.org
[m]
Thank you for your contributions!
I will use this with cwltool
Michael Crusoe
@mr-c:matrix.org
[m]
Melissa Black: If we create a new matrix room for training materials I guess it could be named even more generally like "documentation"? No need for a CWL prefix, I don't think..
melibleq
@melibleq:matrix.org
[m]
ok! documentation and training?
Melissa Black
@melibleq:matrix.org
[m]
alright - so here's the new room for anybody who wants to contribute with those topics! https://matrix.to/#/#docs-training:matrix.org
Michael Crusoe
@mr-c:matrix.org
[m]
The new room is Matrix only; (our existing rooms are mirrored between Gitter & Matrix). To see all the CWL rooms in Matrix, check out https://matrix.to/#/#cwl:matrix.org
Michael Crusoe
@mr-c:matrix.org
[m]
Michael Crusoe
@mr-c:matrix.org
[m]
I haven't seen https://pkp.sfu.ca/ocs/ ; so something to consider later. Do we have a place to dump notes for next year?
Michael Crusoe
@mr-c:matrix.org
[m]
pjotrp: We do plan to rewrite the CWL reference implementation using our autogenerated parser & object model; should be easy to then port the runner onto another language like lisp/scheme/guile; just add codegen support for your favorite language to https://github.com/common-workflow-language/schema_salad/blob/main/schema_salad/codegen.py like has been done for TypeScript and Java recently. Or make your own schema-salad implementation & code generator like @tom-tan is doing for DLang https://github.com/tom-tan/schema-salad-d
pjotrp: if you'd like to see native support for GNU Guix profiles in the CWL reference runner or toil-cwl-runner, all it might take is configuration of Galaxy's dependency resolver ; or at worse case a PR to https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/tool_util/deps
Michael Crusoe
@mr-c:matrix.org
[m]

Re https://hpc.guix.info/blog/2022/01/ccwl-for-concise-and-painless-cwl-workflows/

Each command or step in the workflow needs its own CWL file.

This isn't true. You can use cwlpack from https://github.com/rabix/sbpack to create a single CWL file for an entire workflow (or follow the same style by hand).

1 reply
I'm happy to see additional syntaxes that compile to CWL ; even if a lisp-y/guile syntax is not my normal cup of tea πŸ™‚
Fabrice Brito
@fabricebrito
@mr-c:matrix.org About the conference, we are thinking about putting together a "From zero to CWL on Kubernetes". It's kind of a guide to setup the k8s cluster and run a CWL
4 replies
Michael Crusoe
@mr-c:matrix.org
[m]
Haha, good to know πŸ˜€
Peter Amstutz
@tetron:matrix.org
[m]
on the topic of layering on top of CWL, I did finally document my ProcessGenerator concept https://github.com/common-workflow-language/cwltool/blob/main/docs/processgen.rst
Michael Crusoe
@mr-c:matrix.org
[m]
Melissa Black
@melibleq:matrix.org
[m]
thanks!
Jasper Koehorst
@jjkoehorst_gitlab

Not sure if it is a bug but when i define

  kraken_database:
    type: string > Directory
    doc: absolute path with database location of kraken2

The string type to a Directory it copies the entire directory to a /tmp location

root@1665070d4975:/tmp# head 94k92our/data/8b/8b99471181e23778f8b65acb5c30e1c38eac78db 
>kraken:taxid|254247|NZ_LR699119.1 Aquicella siphonis strain SGT-108 chromosome 1
GCAATTCTGCAAACTATACCCGCTTTCTGTTTGAACCGCCATCCTGATACGGTTTTTTAT

which is going up to 100+gb why is it copying a directory from a local location to a local temp location on the same harddrive?

should i replace all Directory by strings?
this is with β€”provenance
Michael Crusoe
@mr-c:matrix.org
[m]
This is another consequence of common-workflow-language/cwltool#1586 not being implemented; all inputs and intermediates are copied as part of --provenance; and I agree that it needs fixing!
Jasper Koehorst
@jjkoehorst_gitlab
thanks for the comment, is it possible to use string[] as workflow input and map that to File[] in a specific step?