Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 23 22:07
    w-gao edited #4113
  • May 23 22:07
    w-gao edited #4113
  • May 23 22:07
    w-gao edited #4113
  • May 23 21:31
    w-gao edited #4113
  • May 23 21:31
    w-gao edited #4113
  • May 23 21:31
    w-gao edited #4113
  • May 23 21:31
    w-gao edited #4113
  • May 23 21:30
    w-gao edited #4113
  • May 23 21:30
    w-gao edited #4113
  • May 23 21:30
    w-gao edited #4113
  • May 23 21:00
    w-gao edited #4113
  • May 23 19:30
    adamnovak review_requested #4113
  • May 23 18:56
    w-gao synchronize #4052
  • May 23 18:56

    w-gao on 3876-run-cwl-tests-via-wes

    AGC Integration (#4039) * Impl… Stop people asking for ARM Meso… Bump mypy from 0.931 to 0.941 (… and 17 more (compare)

  • May 23 16:29
    adamnovak commented #4110
  • May 23 16:28
    adamnovak reopened #4111
  • May 23 16:27
    adamnovak opened #4113
  • May 23 16:20

    adamnovak on 4109-4094-4107-megabranch

    Merge remote-tracking branch 'u… Bring back psutil (and numpy) (compare)

  • May 23 16:15
    adamnovak reopened #4106
  • May 23 16:13

    adamnovak on master

    (compare)

Michael R. Crusoe
@mr-c:matrix.org
[m]
Sounds like a bug incwltool; it should be possible to make any directory writable
@adrabent: Can you open a bug report at https://github.com/common-workflow-language/cwltool/issues/new ?
1 reply
Douglas Lowe
@douglowe
9 replies
Michael R. Crusoe
@mr-c:matrix.org
[m]
I know that AWS deployments can use Grafana, but I have no experience with that https://toil.readthedocs.io/en/latest/running/cliOptions.html?highlight=grafana#optionsref

https://toil.readthedocs.io/en/latest/running/cloud/amazon.html?highlight=grafana#dashboard

To launch this dashboard for a toil workflow, include the --metrics flag in the toil script command. The dashboard can then be viewed in your browser at localhost:3000

Douglas Lowe
@douglowe
Interesting. I'll have a look at Grafana and see if that gives anything different.
Manabu ISHII
@manabuishii:matrix.org
[m]

Hello.
I'm Manabu Ishii.
I use toil-cwl-runner at several academic institute typical onpremiss cluster.

I have three questions.

Question 1.
I am running toil-cwl-runner in a slurm, shared filesystem, singularity environment.
It runs fine.
My question is that after the execution of toil-cwl-runner, the result file is moved to the shared filesystem, but this seems to be slow.
How is this done? copy or move ? or other way ?

Question 2.

I use toil-cwl-runner is not on AWS.
On-premise, on a shared computer, on a shared file system.
Is it possible to use object storage (s3 compatible) as input files for CWL?

Question 3.
I want to know about toil stats and jobStore relation.
Is there any good documentaion ?

I found this.
https://toil.readthedocs.io/en/latest/running/cloud/clusterUtils.html#cli-status

Thanks nice software !!

adrabent
@adrabent

Hi,
we are using cwltool as runner for our CWL pipelines. Parts of our pipelines need recent bugfixes already merged with current master. Unfortunately, those bugfixes do not seem to be part of a fresh cwltool installation, e.g., via pip, presumably since the master branch has not been tagged for almost two months from now.

Is there any foreseeable timeframe when the developers are planning to "release" a new cwltool version? Thanks already in advance!

Alex

Michael R. Crusoe
@mr-c:matrix.org
[m]
Hello Alex. Do you need toil-cwl-runner to use a newer cwltool version; or do you also need a newer release of cwltool itself?
4 replies
Michael R. Crusoe
@mr-c:matrix.org
[m]
Okay. I'll make a new cwltool release
Douglas Lowe
@douglowe
blog post on one of our RSE's using Toil within the Nvidia Omniverse: https://research-it.manchester.ac.uk/news/2022/03/21/bringing-the-metaverse-to-fusion-research/
Michael R. Crusoe
@mr-c:matrix.org
[m]
Robert Dow
@s8209520
Hi there,
Oops - I am (as you can see) a Toil newbie. I am trying to install it onto our system. I am running a local copy of python 3.10.4. When I run the simple helloworld example from the docs, I am getting this error:
File " [...] /hello.py", line 2, in <module>
from toil.job import Job
File "[...] /toilve/lib/python3.10/site-packages/toil/job.py", line 44, in <module>
from typing_extensions import Literal, overload
ModuleNotFoundError: No module named 'typing_extensions
Michael R. Crusoe
@mr-c:matrix.org
[m]
Can you open a bug report? As a workaround, pip install typing-extensions
Robert Dow
@s8209520
@mr-c:matrix.org Yes, I will. your workaround worked perfectly! Thanks!
1 reply
Marcel Loose
@gmloose
I was trying to use the MPIRequirement to run a CommandLineTool, using the code snippet given here: https://github.com/common-workflow-language/cwltool#running-mpi-based-tools-that-need-to-be-launched. However, this doesn't validate. cwltool --validate returns: INFO /home/marcel/code/rapthor/venv/bin/cwltool 3.1.20211107152837 INFO Resolved 'mpi.cwl' to 'file:///home/marcel/temp/cwl/mpi.cwl' ERROR Tool definition failed validation: mpi.cwl:6:1: checking field `requirements` mpi.cwl:7:3: checking item Field `class` contains undefined reference to `http://commonwl.org/cwltool#MPIRequirement`
6 replies
Michael R. Crusoe
@mr-c:matrix.org
[m]
Correct, this feature is an extension. It isn't present in any of the CWL standards. It could appear in a future version of the CWL standards as a unprefixed optional feature (CWL v1.3 or later)
1 reply
Are you still passing --enable-ext?
If so, then that is a bug
Rohith B S
@rohith-bs
[2022-05-11T06:09:53+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2022-05-11T06:09:53+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host bsr.
[2022-05-11T06:09:53+0000] [MainThread] [I] [toil.worker] Working on job 'normalize_and_index' kind-submit_run/instance-lt36baix v6
[2022-05-11T06:09:53+0000] [MainThread] [I] [numexpr.utils] Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/toil/worker.py", line 379, in workerScript
    job = Job.loadJob(jobStore, jobDesc)
  File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2285, in loadJob
    job = cls._unpickle(userModule, fileHandle, requireInstanceOf=Job)
  File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 1910, in _unpickle
    runnable = unpickler.load()
  File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2904, in __new__
    return cls._resolve(*args)
  File "/usr/local/lib/python3.7/site-packages/toil/job.py", line 2913, in _resolve
    with cls._jobstore.read_file_stream(jobStoreFileID) as fileHandle:
  File "/usr/lib64/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 596, in read_file_stream
    self._check_job_store_file_id(file_id)
  File "/usr/local/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 779, in _check_job_store_file_id
    raise NoSuchFileException(jobStoreFileID)
toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/k/file-4294b849ee3148f087a8efee7183a598/stream' does not exist.
[2022-05-11T06:09:56+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host bsr
Kindly sugges the cause and possible fix for above issue
Marcel Loose
@gmloose
I have an issue with some output files having a wrong name, which only occurs on some machines. One of the outputs has a trailing _2 in its filename. I traced it down to filename conflict resolution in toil/cwl/cwltoil.py around line 860. However, I have no clue what triggers this name conflict, and I also don't understand why this results in my output files being renamed to something unexpected. I'm using Toil 5.6.0 and cwltool 3.1.20211107152837.
Michael R. Crusoe
@mr-c:matrix.org
[m]
That should only happen if there was an existing file with the same name.
Ideally a user would process the CWL output object (the JSON printed to stdout) to determine the output names of a workflow; therefore the names shouldn't matter
Marcel Loose
@gmloose
Well, in this case it does, because it's a file that's part of a complete directory being outputted. The files in that directory represent some kind of database tables, whose names are determined by the tool that read/writes them. So, it definitely is a problem if one of these files gets renamed, even if its new name is properly reflected in the output JSON file.
Michael R. Crusoe
@mr-c:matrix.org
[m]
ah, I see
can you share the JSON output (gist.github.com or privately to me) ?
Marcel Loose
@gmloose
But the more fundamental, and somewhat worrying, question is: why does this problem only occur on some machines. I get the feeling I'm running into some kind of race condition.
Michael R. Crusoe
@mr-c:matrix.org
[m]
agreed
Marcel Loose
@gmloose
I'll try to collect the output JSON, but I have to rerun the workflow.
mr-c
@mr-c:matrix.org
[m]
Thanks
Marcel Loose
@gmloose
OK, I recreated the JSON output file for the workflow with the renamed file (file extension f4_2). Here's the link to gist.github.com: https://gist.github.com/gmloose/8f1c469ff5084cb38d09ec8fc0c7ff30.
Michael R. Crusoe
@mr-c:matrix.org
[m]
huh. Lines 972 & 1969 are the relevant sections
Marcel Loose
@gmloose
Sure, but what should I write in there? I'm not sure what the bug description should be, other than the name conflict.
Michael R. Crusoe
@mr-c:matrix.org
[m]
That there is a false positive naming conflict ; yes there are two files with the same name, but they are in different directories
Marcel Loose
@gmloose
Marcel Loose
@gmloose
What's the best way to redirect to a file the JSON string containing the outputs generated by a workflow, which is printed to stdout?
Michael R. Crusoe
@mr-c:matrix.org
[m]
toil-cwl-runner ... > outputs.json or similar
Marcel Loose
@gmloose
And if I invoke toil-cwl-runner from a Python script, using toil.cwl.cwltoil.main() function? Instead of redirecting stdout in the Python call, could I just add a line to the CWL Workflow that redirects stdout to a file. Or would that produce more output than just the JSON output?
1 reply
Marcel Loose
@gmloose
Any idea if DataBiosphere/toil#4101 can be picked up soon? It is blocking us, because Toil currently produces bogus results for our workflows. To me this seems to be a serious bug!
3 replies
Michael R. Crusoe
@mr-c:matrix.org
[m]
A repeatable example, but I know that is difficult
What if I gave you a script to fix the renames post execution?
1 reply
Michael R. Crusoe
@mr-c:matrix.org
[m]
Huh
Marcel Loose
@gmloose

Not sure if it's in any way related to #4101, but I have the following expression to convert an array of an array of directories into an array of directories:

id: merge_array_directories
label: merge_array_directories
class: ExpressionTool

cwlVersion: v1.2
inputs:
    - id: input
      type:
        - type: array
          items:
            - type: array
              items: Directory
outputs:
    - id: output
      type: Directory[]

expression: |
  ${
    var out_directory = []
    for(var i=0; i<inputs.input.length; i++){
        var item = inputs.input[i]
        if(item != null){
            out_directory = out_directory.concat(item)
        }
    }
    return {'output': out_directory}
  }

requirements:
  - class: InlineJavascriptRequirement

This expression, when run, produces a number of warnings like:

Warning: invalid field `nameroot`, expected one of: 'class', 'location', 'path', 'basename', 'listing'
[2022-05-20T18:04:18+0200] [MainThread] [W] [salad] Warning: invalid field `nameroot`, expected one of: 'class', 'location', 'path', 'basename', 'listing'
2 replies
Michael R. Crusoe
@mr-c:matrix.org
[m]
Do you get the same warnings when running just the ExpressionTool? Is this is batch or single machine mode?
1 reply
Michael R. Crusoe
@mr-c:matrix.org
[m]
There is a scatter prior to this step?
2 replies
Okay. If you can run with debug worker and the regular debug, that should show the input objects prior to this step, and will provide a big hint as to what is going on. I'll need the entire log
3 replies