@karma29:matrix.org yes that would be appropriate. also https://gitter.im/common-workflow-language/common-workflow-language
hello! just curious about the code here : https://github.com/DataBiosphere/toil/blob/master/src/toil/cwl/cwltoil.py#L736
mutable is by default set to false, which means that there is no downloaded copy of the file (and a link to it is created instead). in the function call examples, there wasn't any explicit declaration of mutable to true, so do they exhibit "streamable" properties? what more changes should we make here?
actually i'm a gsoc'21 applicant so i wasn't sure if it would be more suitable to reply to the comment threads on github or join the irc. what's recommended though? i'm fine with either! 😃
[Adam Novak, UCSC GI] I think the idea behind CWL
streamable is that you will get a pipe (a FIFO) presented to the tool instead of a normal file? I'm not really sure.
[Adam Novak, UCSC GI] Toil already has a
file_store.readGlobalFileStream function, but that returns a Python file object, and doesn't currently produce a FIFO on disk. So I think getting CWL streamable support would involve handling streamable requests from CWL by using file_store.readGlobalFileStream and producing a FIFO and a thread to fill it with data.
okay, i see! thank you 👍️ so we need to implement a FIFO process for
- a sequence of files (irrespective of whether they're downloaded or not, because i think right now it waits for all the input files to be processed to release the outputs?)
- a sequence of data within the file, in case the file comes from a remote location or s3 url (so we don't need to download it)
- combination of the above two, if there are more files from a remote location or s3 url in the pipe
is that correct?
I'll try to check the gitter manually more often.
Thanks!
1 reply
Okay, here's the doc link crusoe @DailyDreaming https://docs.google.com/document/d/1OciTGIqblEHK5qHKDk6e7C4Pbo3UTszdxyvZSVwpL7g/edit?usp=sharing
I'd be grateful if you could give feedback on it! I'm a bit confused regarding how we would actually implement streamable properties, except for the part where we change
file_store.readGlobalFileStream so any inputs regarding that would be much appreciated
Also is a "job class" a Python class? I have heard of text files getting pickled. So does a job class getting pickled mean that the job class is being implemented as an object and then their contents being pickled to a file?
Okay, thank you for the feedback!
I've made some changes - if it looks good, may I go ahead and submit the proposal?
Is there any interest in my suggestion here: https://github.com/DataBiosphere/toil/issues/1768#issuecomment-818475173 ? I'd be happy to give it a go if it's okayed by a maintainer