Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jun 21 15:04
    w-gao synchronize #3488
  • Jun 21 15:04

    w-gao on 3487-migrate-cloudconfig-to-ignition

    Bring back cwl test (compare)

  • Jun 21 13:19
    mr-c commented #3652
  • Jun 21 08:06
    jonathanxu18 review_requested #3659
  • Jun 21 08:06
    jonathanxu18 review_requested #3659
  • Jun 21 08:06
    jonathanxu18 review_requested #3657
  • Jun 21 08:06
    jonathanxu18 review_requested #3657
  • Jun 21 08:06
    jonathanxu18 review_requested #3656
  • Jun 21 08:06
    jonathanxu18 review_requested #3656
  • Jun 21 07:33
    jonathanxu18 opened #3659
  • Jun 21 07:31

    jonathanxu18 on 3658-add-type-hints-abstractJobStore.py

    Add type hints and comments (compare)

  • Jun 21 07:19
    unito-bot edited #3658
  • Jun 21 07:18
    jonathanxu18 opened #3658
  • Jun 21 07:17
    jonathanxu18 opened #3657
  • Jun 21 07:16

    jonathanxu18 on 3631-add-type-hints-toilDebugFile.py

    Add type hints (compare)

  • Jun 21 07:12
    jonathanxu18 opened #3656
  • Jun 21 07:12

    jonathanxu18 on 3630-add-type-hints-toilLaunchCluster.py

    Remove new line (compare)

  • Jun 21 07:09
    jonathanxu18 edited #3630
  • Jun 21 07:09

    jonathanxu18 on 3630-add-type-hints-toilLaunchCluster.py

    Add type hints (compare)

  • Jun 21 07:05
    jonathanxu18 closed #3623
karma29
@karma29:matrix.org
[m]
actually i'm a gsoc'21 applicant so i wasn't sure if it would be more suitable to reply to the comment threads on github or join the irc. what's recommended though? i'm fine with either! 😃
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] We watch Github issues, and the chat here, but not really Github code comments. Either there or here is fine; here is maybe better fro questions that are not themselves bugs.
[Adam Novak, UCSC GI] I think the idea behind CWL streamable is that you will get a pipe (a FIFO) presented to the tool instead of a normal file? I'm not really sure.
Peter Amstutz
@tetron
yes
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] Anyway, readGlobalFile produces an ordinary file, with a filename and all the data on disk. It might be given via a symlink, and the lack of mutable means the user code isn't allowed to modify it, as other jobs may be using the same copy.
Peter Amstutz
@tetron
and if the data is coming from a remote location, it can be streamed incrementally instead of waiting for a full download
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] Toil already has a file_store.readGlobalFileStream function, but that returns a Python file object, and doesn't currently produce a FIFO on disk. So I think getting CWL streamable support would involve handling streamable requests from CWL by using file_store.readGlobalFileStream and producing a FIFO and a thread to fill it with data.
Michael R. Crusoe
@mr-c
Or by using a third party utility to achieve the same thing, given the s3 url
karma29
@karma29:matrix.org
[m]

okay, i see! thank you 👍️ so we need to implement a FIFO process for

  1. a sequence of files (irrespective of whether they're downloaded or not, because i think right now it waits for all the input files to be processed to release the outputs?)
  2. a sequence of data within the file, in case the file comes from a remote location or s3 url (so we don't need to download it)
  3. combination of the above two, if there are more files from a remote location or s3 url in the pipe

is that correct?

Lon Blauvelt
@DailyDreaming
@karma29:matrix.org I'm sorry, it seems the matrix.org connection doesn't sync to our slack channel, so your message was missed. In general, this will need to be one FIFO per file. I would focus on the AWS s3 functionality alone first.
I'll try to check the gitter manually more often.
Michael R. Crusoe
@mr-c
@DailyDreaming does your slack connection sync via gitter? Maybe switch to directly syncing your slack channel with the matrix version?
Lon Blauvelt
@DailyDreaming
I'll try that.
Thanks!
karma29
@karma29:matrix.org
[m]
Okay! Is there a link to the slack group i can join?
1 reply
I'd be grateful if you could give feedback on it! I'm a bit confused regarding how we would actually implement streamable properties, except for the part where we change file_store.readGlobalFileStream so any inputs regarding that would be much appreciated
karma29
@karma29:matrix.org
[m]
Also is a "job class" a Python class? I have heard of text files getting pickled. So does a job class getting pickled mean that the job class is being implemented as an object and then their contents being pickled to a file?
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] The jobs in Toil are instances of various "job classes", like CWLJob and JobFunctionWrappingJob. We do indeed instantiate these classes and then pickle the resulting objects into files.
karma29
@karma29:matrix.org
[m]

Okay, thank you for the feedback!

I've made some changes - if it looks good, may I go ahead and submit the proposal?

karma29
@karma29:matrix.org
[m]

Hello!

Just a follow-up to my previous message, since there's ~ 2 hours for the submission to end 😃 crusoe @DailyDreaming

1 reply
karma29
@karma29:matrix.org
[m]
Thank you, just updated with another query. Can you please check the doc? Thank you crusoe @DailyDreaming
Dennis R Kennetz
@drkennetz
Is it appropriate to post job openings in here? We have a more senior position open that is CWL/toil related for bioinformatics.
Michael Milton
@TMiguelT
Is there any interest in my suggestion here: https://github.com/DataBiosphere/toil/issues/1768#issuecomment-818475173 ? I'd be happy to give it a go if it's okayed by a maintainer
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] It sounds like a good idea to me!
Lon Blauvelt
@DailyDreaming
@drkennetz That's fine by me.
crusoe
@mr-c:matrix.org
[m]
Google just "alerted" me to this 11 month old toil-cwl-runner question on biostars https://www.biostars.org/p/448085
Lon Blauvelt
@DailyDreaming
@mr-c:matrix.org Thanks for the link! Good work around hack, I agree. We should definitely make it less hacky. I'll create a Toil-side issue.
mareq
@mareq
hello, i am not sure if this is the correct place to ask but anyway: i would like to run cwl workflows in azure cloud and toil seems like one of the good ways to go. the k8s option looks quite good (based on the documentation i have read), except it says that it requires aws for job store. hence the question: is there some other way (possibly without even k8s: it would be great to do it in a way independent of particular cloud, but it is not hard requirement) to use toil for running cwl workflows on azure and azure only?
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] @mareq, we used to have an Azure job store and an Azure template for stamping out clusters, but we dropped them. The AWS job store is not able to run on top of just any S3-compatible storage system; we currently also need SimpleDB, and we're swapping that out for DynamoDB which also isn't part of S3.
[Adam Novak, UCSC GI] In the future to run on Azure we would use the Kubernetes batch system, because everybody has a good way to provide Kubernetes, but there's still the need for a job store. The only one that isn't tied to a cloud provider is the one that requires a shared filesystem.
[Adam Novak, UCSC GI] We might be able to get the file job store working with Kubernetes if you can get all your nodes a shared strongly-consistent filesystem (which NFS sort of isn't) and we can get it exposed to the Toil kubernetes pods, maybe with some kind of extra host path volume.
[Adam Novak, UCSC GI] But current;y we don't have a way to do what you want out of the box, unfortunately.
mareq
@mareq
@DailyDreaming throwing in additional assumptions: the workflows i need to run are tiny and the data they process in order of MBs for each job, so i do not really need super-distributed system (performance is not an issue at all). i just need some cwl-running backend to put behind the azure-based webapp, which allows its users run cwl workflows one job at a time (the webapp itself is multi-user, so there will be parallelism coming from there, but i guess that is just an irrelevant detail for the purpose of this discussion); based on your answer, this would fly with some sort of filesystem job store, right?
Lon Blauvelt
@DailyDreaming
@mareq If you want to just run toil scripts as single_machine on an Azure instance, that would work. But we no longer have any native Azure support (dropped due to funding): DataBiosphere/toil#2860
mareq
@mareq
i see. thank you :thumbsup:
Peter Amstutz
@tetron
hi @mareq for running CWL workflows on Azure another option is http://arvados.org
Douglas Lowe
@douglowe
I'm testing a cwl workflow with toil and slurm, and finding that my runtime is being dominated by queue wait times, as a lot of the tasks are preparatory work for the main task, any don't take a lot of time to run. Is there a way of asking cwl-toil to submit tasks together within the same job, or (as I'm running the toil job manager in it's own serial job) can I mark jobs that should be run within the primary job (I guess the reverse of the intent of --runCwlInternalJobsOnWorkers)?
Lon Blauvelt
@DailyDreaming
[Adam Novak, UCSC GI] Hm. Toil has a "chaining" feature where single direct child or follow-on jobs with the same resource requirements will run in the same execution as the previous job. If you set your resource requirements so the child jobs are no larger than the parent ones, you might be able to use that.
[Adam Novak, UCSC GI] To get stuff to run on the Toil leader, right now we match on particular internal job names for jobs that are part of Toil's CWL interpreter machinery. We don't have a feature currently to let you add other particular jobs to that set.
Douglas Lowe
@douglowe
Possibly the 'chaining' feature might be the best approach for me to try. I'll try explicitly setting resource requirements for some of the child jobs, and see this cause them to share executions.
Ian
@ionox0
Would anyone have any guess as to how I might be getting file prefixes that look like _:file:///... in my workflow?
    FileNotFoundError: [Errno 2] No such file or directory: '_:file:///juno/work/access/testing/users/johnsoni/access_qc/work5/f26b1779cb2f5ea39419d80a2066faf9/e81b/360a/tmpb4cbwihh/out/simplex_bam_pool_a_dir'
I'm on toil 5.4
most of them look like
toilfs:16305:0:files/for-job/kind-CWLJob/instance-7ousmklr/file-184521b84be2407b8ea6ef01398c6f81/histogram.pdf
Ian
@ionox0
but some of them, specifically Directory objects, have the _:file:// prefix
Lon Blauvelt
@DailyDreaming

[Adam Novak, UCSC GI] @ionox0 This is part of the not-actually-correct logic that Toil has for preparing directories for CWL workflows. The underscore prefix is something that cwltool uses that indicates a directory is to be created to present to a CWL tool. Toil tries to generate this to control cwltool, but in practice what we have in 5.4 only works when running on a single machine or otherwise using a shared filesystem between nodes.

I've been redoing all that logic in DataBiosphere/toil#3628 so that Toil can be responsible for setting up the directory structures that CWL tools expect to see, whether there's a shared filesystem or not, but I still don't have it fully working yet. When it's done, it should be much harder to break.

Ian
@ionox0
Thanks for the quick response, that makes sense, I'm getting this issue on a shared filesystem, but perhaps our workflow is a bit complicated in this case because I'm dealing with a Directory object that also has Directory objects inside of it.
For now would you suggest us avoiding the use of Directory objects in 5.4 ?