adamnovak on 2501-file-store-case-insensitive
Performance testing for Gravito… Merge branch 'master' into issu… (compare)
hello! just curious about the code here : https://github.com/DataBiosphere/toil/blob/master/src/toil/cwl/cwltoil.py#L736
mutable is by default set to false, which means that there is no downloaded copy of the file (and a link to it is created instead). in the function call examples, there wasn't any explicit declaration of mutable to true, so do they exhibit "streamable" properties? what more changes should we make here?
streamableis that you will get a pipe (a FIFO) presented to the tool instead of a normal file? I'm not really sure.
file_store.readGlobalFileStreamfunction, but that returns a Python file object, and doesn't currently produce a FIFO on disk. So I think getting CWL streamable support would involve handling streamable requests from CWL by using
file_store.readGlobalFileStreamand producing a FIFO and a thread to fill it with data.
okay, i see! thank you 👍️ so we need to implement a FIFO process for
is that correct?
file_store.readGlobalFileStreamso any inputs regarding that would be much appreciated
Okay, thank you for the feedback!
I've made some changes - if it looks good, may I go ahead and submit the proposal?
FileNotFoundError: [Errno 2] No such file or directory: '_:file:///juno/work/access/testing/users/johnsoni/access_qc/work5/f26b1779cb2f5ea39419d80a2066faf9/e81b/360a/tmpb4cbwihh/out/simplex_bam_pool_a_dir'
[Adam Novak, UCSC GI] @ionox0 This is part of the not-actually-correct logic that Toil has for preparing directories for CWL workflows. The underscore prefix is something that
cwltool uses that indicates a directory is to be created to present to a CWL tool. Toil tries to generate this to control
cwltool, but in practice what we have in 5.4 only works when running on a single machine or otherwise using a shared filesystem between nodes.
I've been redoing all that logic in DataBiosphere/toil#3628 so that Toil can be responsible for setting up the directory structures that CWL tools expect to see, whether there's a shared filesystem or not, but I still don't have it fully working yet. When it's done, it should be much harder to break.