w-gao on 3876-run-cwl-tests-via-wes
AGC Integration (#4039) * Impl… Stop people asking for ARM Meso… Bump mypy from 0.931 to 0.941 (… and 17 more (compare)
adamnovak on 4109-4094-4107-megabranch
Merge remote-tracking branch 'u… Bring back psutil (and numpy) (compare)
adamnovak on master
cwltool; it should be possible to make any directory writable
To launch this dashboard for a toil workflow, include the
--metricsflag in the toil script command. The dashboard can then be viewed in your browser at localhost:3000
I'm Manabu Ishii.
I use toil-cwl-runner at several academic institute typical onpremiss cluster.
I have three questions.
I am running toil-cwl-runner in a slurm, shared filesystem, singularity environment.
It runs fine.
My question is that after the execution of toil-cwl-runner, the result file is moved to the shared filesystem, but this seems to be slow.
How is this done? copy or move ? or other way ?
I use toil-cwl-runner is not on AWS.
On-premise, on a shared computer, on a shared file system.
Is it possible to use object storage (s3 compatible) as input files for CWL?
I want to know about toil stats and jobStore relation.
Is there any good documentaion ?
Thanks nice software !!
we are using cwltool as runner for our CWL pipelines. Parts of our pipelines need recent bugfixes already merged with current master. Unfortunately, those bugfixes do not seem to be part of a fresh cwltool installation, e.g., via pip, presumably since the master branch has not been tagged for almost two months from now.
Is there any foreseeable timeframe when the developers are planning to "release" a new cwltool version? Thanks already in advance!
toil-cwl-runnerto use a newer
cwltoolversion; or do you also need a newer release of
pip install typing-extensions
MPIRequirementto run a
CommandLineTool, using the code snippet given here: https://github.com/common-workflow-language/cwltool#running-mpi-based-tools-that-need-to-be-launched. However, this doesn't validate.
INFO /home/marcel/code/rapthor/venv/bin/cwltool 3.1.20211107152837 INFO Resolved 'mpi.cwl' to 'file:///home/marcel/temp/cwl/mpi.cwl' ERROR Tool definition failed validation: mpi.cwl:6:1: checking field `requirements` mpi.cwl:7:3: checking item Field `class` contains undefined reference to `http://commonwl.org/cwltool#MPIRequirement`
Kindly sugges the cause and possible fix for above issue
[2022-05-11T06:09:53+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-05-11T06:09:53+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host bsr. [2022-05-11T06:09:53+0000] [MainThread] [I] [toil.worker] Working on job 'normalize_and_index' kind-submit_run/instance-lt36baix v6 [2022-05-11T06:09:53+0000] [MainThread] [I] [numexpr.utils] Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/toil/worker.py", line 379, in workerScript job = Job.loadJob(jobStore, jobDesc) File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2285, in loadJob job = cls._unpickle(userModule, fileHandle, requireInstanceOf=Job) File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 1910, in _unpickle runnable = unpickler.load() File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2904, in __new__ return cls._resolve(*args) File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2913, in _resolve with cls._jobstore.read_file_stream(jobStoreFileID) as fileHandle: File "/usr/lib64/python3.7/contextlib.py", line 112, in __enter__ return next(self.gen) File "/usr/local/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 596, in read_file_stream self._check_job_store_file_id(file_id) File "/usr/local/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 779, in _check_job_store_file_id raise NoSuchFileException(jobStoreFileID) toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/k/file-4294b849ee3148f087a8efee7183a598/stream' does not exist. [2022-05-11T06:09:56+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host bsr
_2in its filename. I traced it down to filename conflict resolution in
toil/cwl/cwltoil.pyaround line 860. However, I have no clue what triggers this name conflict, and I also don't understand why this results in my output files being renamed to something unexpected. I'm using Toil 5.6.0 and cwltool 3.1.20211107152837.
f4_2). Here's the link to gist.github.com: https://gist.github.com/gmloose/8f1c469ff5084cb38d09ec8fc0c7ff30.
toil-cwl-runnerfrom a Python script, using
toil.cwl.cwltoil.main()function? Instead of redirecting
stdoutin the Python call, could I just add a line to the CWL Workflow that redirects
stdoutto a file. Or would that produce more output than just the JSON output?
Not sure if it's in any way related to #4101, but I have the following expression to convert an array of an array of directories into an array of directories:
This expression, when run, produces a number of warnings like:
Warning: invalid field `nameroot`, expected one of: 'class', 'location', 'path', 'basename', 'listing' [2022-05-20T18:04:18+0200] [MainThread] [W] [salad] Warning: invalid field `nameroot`, expected one of: 'class', 'location', 'path', 'basename', 'listing'