hello! just curious about the code here : https://github.com/DataBiosphere/toil/blob/master/src/toil/cwl/cwltoil.py#L736
mutable is by default set to false, which means that there is no downloaded copy of the file (and a link to it is created instead). in the function call examples, there wasn't any explicit declaration of mutable to true, so do they exhibit "streamable" properties? what more changes should we make here?
actually i'm a gsoc'21 applicant so i wasn't sure if it would be more suitable to reply to the comment threads on github or join the irc. what's recommended though? i'm fine with either! 😃
[Adam Novak, UCSC GI] I think the idea behind CWL
streamable is that you will get a pipe (a FIFO) presented to the tool instead of a normal file? I'm not really sure.
[Adam Novak, UCSC GI] Toil already has a
file_store.readGlobalFileStream function, but that returns a Python file object, and doesn't currently produce a FIFO on disk. So I think getting CWL streamable support would involve handling streamable requests from CWL by using file_store.readGlobalFileStream and producing a FIFO and a thread to fill it with data.
okay, i see! thank you 👍️ so we need to implement a FIFO process for
- a sequence of files (irrespective of whether they're downloaded or not, because i think right now it waits for all the input files to be processed to release the outputs?)
- a sequence of data within the file, in case the file comes from a remote location or s3 url (so we don't need to download it)
- combination of the above two, if there are more files from a remote location or s3 url in the pipe
is that correct?
I'll try to check the gitter manually more often.
Thanks!
1 reply
Okay, here's the doc link crusoe @DailyDreaming https://docs.google.com/document/d/1OciTGIqblEHK5qHKDk6e7C4Pbo3UTszdxyvZSVwpL7g/edit?usp=sharing
I'd be grateful if you could give feedback on it! I'm a bit confused regarding how we would actually implement streamable properties, except for the part where we change
file_store.readGlobalFileStream so any inputs regarding that would be much appreciated
Also is a "job class" a Python class? I have heard of text files getting pickled. So does a job class getting pickled mean that the job class is being implemented as an object and then their contents being pickled to a file?
Okay, thank you for the feedback!
I've made some changes - if it looks good, may I go ahead and submit the proposal?
Is there any interest in my suggestion here: https://github.com/DataBiosphere/toil/issues/1768#issuecomment-818475173 ? I'd be happy to give it a go if it's okayed by a maintainer
Google just "alerted" me to this 11 month old toil-cwl-runner question on biostars https://www.biostars.org/p/448085
hello, i am not sure if this is the correct place to ask but anyway: i would like to run cwl workflows in azure cloud and toil seems like one of the good ways to go. the k8s option looks quite good (based on the documentation i have read), except it says that it requires aws for job store. hence the question: is there some other way (possibly without even k8s: it would be great to do it in a way independent of particular cloud, but it is not hard requirement) to use toil for running cwl workflows on azure and azure only?
[Adam Novak, UCSC GI] @mareq, we used to have an Azure job store and an Azure template for stamping out clusters, but we dropped them. The AWS job store is not able to run on top of just any S3-compatible storage system; we currently also need SimpleDB, and we're swapping that out for DynamoDB which also isn't part of S3.
[Adam Novak, UCSC GI] In the future to run on Azure we would use the Kubernetes batch system, because everybody has a good way to provide Kubernetes, but there's still the need for a job store. The only one that isn't tied to a cloud provider is the one that requires a shared filesystem.
[Adam Novak, UCSC GI] We might be able to get the file job store working with Kubernetes if you can get all your nodes a shared strongly-consistent filesystem (which NFS sort of isn't) and we can get it exposed to the Toil kubernetes pods, maybe with some kind of extra host path volume.
[Adam Novak, UCSC GI] But current;y we don't have a way to do what you want out of the box, unfortunately.
@DailyDreaming throwing in additional assumptions: the workflows i need to run are tiny and the data they process in order of MBs for each job, so i do not really need super-distributed system (performance is not an issue at all). i just need some cwl-running backend to put behind the azure-based webapp, which allows its users run cwl workflows one job at a time (the webapp itself is multi-user, so there will be parallelism coming from there, but i guess that is just an irrelevant detail for the purpose of this discussion); based on your answer, this would fly with some sort of filesystem job store, right?
@mareq If you want to just run toil scripts as single_machine on an Azure instance, that would work. But we no longer have any native Azure support (dropped due to funding): DataBiosphere/toil#2860
I'm testing a cwl workflow with toil and slurm, and finding that my runtime is being dominated by queue wait times, as a lot of the tasks are preparatory work for the main task, any don't take a lot of time to run. Is there a way of asking cwl-toil to submit tasks together within the same job, or (as I'm running the toil job manager in it's own serial job) can I mark jobs that should be run within the primary job (I guess the reverse of the intent of --runCwlInternalJobsOnWorkers)?
[Adam Novak, UCSC GI] Hm. Toil has a "chaining" feature where single direct child or follow-on jobs with the same resource requirements will run in the same execution as the previous job. If you set your resource requirements so the child jobs are no larger than the parent ones, you might be able to use that.
[Adam Novak, UCSC GI] To get stuff to run on the Toil leader, right now we match on particular internal job names for jobs that are part of Toil's CWL interpreter machinery. We don't have a feature currently to let you add other particular jobs to that set.
FileNotFoundError: [Errno 2] No such file or directory: '_:file:///juno/work/access/testing/users/johnsoni/access_qc/work5/f26b1779cb2f5ea39419d80a2066faf9/e81b/360a/tmpb4cbwihh/out/simplex_bam_pool_a_dir'
I'm on toil 5.4
most of them look like
toilfs:16305:0:files/for-job/kind-CWLJob/instance-7ousmklr/file-184521b84be2407b8ea6ef01398c6f81/histogram.pdf