Hexotical on 3514-missing-permission-warning-aws
Performance testing for Gravito… Issues/2501 file store case ins… cache results of cwltool versio… and 7 more (compare)
Hexotical on 3514-missing-permission-warning-aws
Fix iam.py to pass tests (compare)
Hexotical on 3514-missing-permission-warning-aws
Type iam.py to pass mypy check (compare)
Hexotical on 3514-missing-permission-warning-aws
Remove unused import in aws pro… (compare)
Hexotical on 3514-missing-permission-warning-aws
Address PR comments, remove uti… (compare)
mr-c on 3131-do-not-submit-empty-jobs
Update leader.py (compare)
file_store.readGlobalFileStream
function, but that returns a Python file object, and doesn't currently produce a FIFO on disk. So I think getting CWL streamable support would involve handling streamable requests from CWL by using file_store.readGlobalFileStream
and producing a FIFO and a thread to fill it with data.
okay, i see! thank you 👍️ so we need to implement a FIFO process for
is that correct?
file_store.readGlobalFileStream
so any inputs regarding that would be much appreciated
Okay, thank you for the feedback!
I've made some changes - if it looks good, may I go ahead and submit the proposal?
FileNotFoundError: [Errno 2] No such file or directory: '_:file:///juno/work/access/testing/users/johnsoni/access_qc/work5/f26b1779cb2f5ea39419d80a2066faf9/e81b/360a/tmpb4cbwihh/out/simplex_bam_pool_a_dir'
toilfs:16305:0:files/for-job/kind-CWLJob/instance-7ousmklr/file-184521b84be2407b8ea6ef01398c6f81/histogram.pdf
[Adam Novak, UCSC GI] @ionox0 This is part of the not-actually-correct logic that Toil has for preparing directories for CWL workflows. The underscore prefix is something that cwltool
uses that indicates a directory is to be created to present to a CWL tool. Toil tries to generate this to control cwltool
, but in practice what we have in 5.4 only works when running on a single machine or otherwise using a shared filesystem between nodes.
I've been redoing all that logic in DataBiosphere/toil#3628 so that Toil can be responsible for setting up the directory structures that CWL tools expect to see, whether there's a shared filesystem or not, but I still don't have it fully working yet. When it's done, it should be much harder to break.
exit code 120
pointing to ?