by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Kaushik Ghose
    @kaushik-work
    @mr-c , unfortunately that doesn't seem to be the issue. Would you have any other HPC + Toil tips?
    Any way to get more detailed logs?
    Also, any way to test Toil's LSF backend without CWL, to start eliminating where issues are?
    Michael R. Crusoe
    @mr-c
    @kaushik-work What's the current error? Can you manually invoke the _toil_runner as a scheduled job? (Maybe a permissions issue?)
    Kaushik Ghose
    @kaushik-work
    @mr-c we just ran the basic pure Toil example and it ran fine on LSF. So, somehow the CWL is causing issues
    The CWL is very simple, and runs fine in non bacth mode.
    Kaushik Ghose
    @kaushik-work
    @mr-c , we were forgetting to pass a job store ... now it works.
    Michael R. Crusoe
    @mr-c
    @kaushik-work glad you fixed it!
    Ian
    @ionox0
    @kaushik-work sorry I missed this but we use toil heavily with lsf, our configuration is also to do something like bsub toil-cwl-runner except we actually have another python wrapper in between bsub and the toil command
    You have to make sure you are inside of the virtualenv (we actually use conda) that has toil installed when you do the bsub
    lsf will propagate the PATH variable and any other necessary env vars (we also use PYTHONPATH but I'm not sure if that's necessary) through the bsub command, and then we use --preserve-env-vars for those variables as well to pass them to toil
    Kaushik Ghose
    @kaushik-work
    Thank you @ionox0 ! Does calling toil-cwl-runner directly via bsub not negate the benefits of using toils batch scheduling system, and force all jobs in the workflow to run on the same node?
    Michael R. Crusoe
    @mr-c
    They may be able to schedule jobs from within the initially scheduled job.
    Kaushik Ghose
    @kaushik-work
    @mr-c interesting - I thought if you don't use the batch subsystem toil basically operates on a single node (local) mode.
    (So yes, the main toil job gets scheduled by the LSF, but then it just runs on that first node the LSF put it on)
    Ian
    @ionox0
    We submit the toil-cwl-runner command to a specific designated "leader" node and "leader" queue on our batch system, which are designed for long running jobs
    The toil-cwl-runner process is still able to submit child jobs, which will be assigned to any random available nodes
    (if using --batchSystem lsf) we are still using the batch scheduling functionality of toil
    it's a bit strange sounding, but jobs submitted with bsub are also able to submit their own jobs
    Michael R. Crusoe
    @mr-c
    ^^ this is a typical setup once people get comfortable with toil-cwl-runner, and it is nicer to your admins as it reduces the load on the log in nodes.
    Ian
    @ionox0
    yes exact;ly
    Kaushik Ghose
    @kaushik-work
    @ionox0 A very interesting. In my mental model submitting a program a.out to bsub results in the LSF finding an appropriate node for a.out and then running it on that node. How does bsub know what jobs inside a.out to schedule on additional nodes? Is it whenever it sees a.out fork?
    Ian
    @ionox0
    well, toil is essentially calling bsub under the hood, so thats when it knows to "fork", although I'm not sure if you mean forking as in submitting new jobs or some new subprocesses
    the child jobs that toil submits go through the same queueing and scheduling as the original leader job, just with different requirements
    Kaushik Ghose
    @kaushik-work
    I thought Toil only called bsub when the --batchSystem lsf is invoked?
    Ian
    @ionox0
    that's right, we're using that parameter
    i might be misunderstanding the question here
    Kaushik Ghose
    @kaushik-work
    Ah, ok, mystery solved for me. I've been invoking the initial toil command without bsub since I'm fine running the toil scheduler itself on some random node.
    I misunderstood your comment to mean that you invoked toil with bsub but did not use the --batchSystem LSF and my mental model was broken.
    Ian
    @ionox0
    ah got it got it
    Kaushik Ghose
    @kaushik-work
    :laughing: thanks @ionox0 . How are you doing?
    Ian
    @ionox0
    well!! :)
    Kaushik Ghose
    @kaushik-work
    :thumbsup:
    Ian
    @ionox0
    not in NYC at the moment, but some of us may be called "back into duty"
    Kaushik Ghose
    @kaushik-work
    Oh, that is concerning. What kind of PPE will you get? How will you commute?
    Ian
    @ionox0
    it's all volunteer at the moment, and I'm sure they provide all of that, still trying to figure it out but will keep yall updated!
    Kaushik Ghose
    @kaushik-work
    :thumbsup:
    Kaushik Ghose
    @kaushik-work
    Folks, Toil is now Py3 only correct? I can put in Py3 only code?
    Specifically, may I use the new Py3 format string f"{var}"
    Michael R. Crusoe
    @mr-c
    @kaushik-work Yep! python3.6; when did f strings come out?
    Kaushik Ghose
    @kaushik-work
    \o/ Thanks @mr-c
    Vijay Lakhujani
    @lakhujanivijay
    Hi , I am a beginner with Toil but comfortable with Python. I am building a pipeline where I am required to run fastp for a couple for Illumina paired end files. I am using singularity container with toil - already build and executed bwa with that. So for fastp I have downloaded image from singularity hub and build a container
    how to execute any tools which assumes paired end files as input ?
    Kaushik Ghose
    @kaushik-work
    Does cwl-toil-runner support singularity. AFAIK cwltool does.
    Michael R. Crusoe
    @mr-c
    Yes
    --singularity 👍
    Kaushik Ghose
    @kaushik-work
    :thumbsup:
    JaroAmsterdam
    @JaroAmsterdam
    Hi everyone, quick question, to use Toil in combination with AWS, does it matter which version of the AWS CLI I install?
    Lon Blauvelt
    @DailyDreaming
    @JaroAmsterdam You might have problems with very old versions, but the latest should work. If you run into any issues with newer versions, please post an issue since that would definitely be a bug.