Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Mar 31 20:34
    avsastry opened #1549
  • Mar 31 09:54
    genesolutions opened #1548
  • Mar 30 06:38
    pditommaso commented #1535
  • Mar 30 02:47
    marcodelapierre commented #1535
  • Mar 28 18:54
    pditommaso milestoned #1537
  • Mar 28 17:31
    pditommaso commented #1545
  • Mar 28 17:30
    pditommaso closed #1545
  • Mar 28 16:42

    pditommaso on v20.03.0-edge

    (compare)

  • Mar 28 11:21

    pditommaso on master

    Add changelog [release 20.03.0-edge] Update t… (compare)

  • Mar 28 10:33
    pditommaso closed #1547
  • Mar 28 10:33
    pditommaso commented #1547
  • Mar 28 10:31

    pditommaso on master

    Improve singularity warn message (compare)

  • Mar 28 09:54
    pditommaso milestoned #1535
  • Mar 28 09:51
    pditommaso closed #1535
  • Mar 28 09:51
    pditommaso commented #1535
  • Mar 28 09:11
    pditommaso closed #1544
  • Mar 28 09:10
    pditommaso commented #1546
  • Mar 28 09:10
    pditommaso closed #1546
  • Mar 28 09:10
    pditommaso commented #1546
  • Mar 28 08:54

    pditommaso on master

    Add support for GLS private add… (compare)

Stephen Kelly
@stevekm

@idot

    File "/usr/lib64/python3.7/tempfile.py", line 258, in _mkstemp_inner
     fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/dev/shm/pym-29180-mpuki4mm'

obv it seems that the permissions on this file are set to 600, which I belive is user read/write only, but it appears to be a library of some kind and so it might need 700 permissions e.g. read/write/execute. Also, its in your /dev/shm shared memory, was that intentional? Either way you should check on how this file is getting to be loaded in this location and prob do a chmod u+x on it

Paolo Di Tommaso
@pditommaso
@stevekm yes, you are right, feel free to open an issue as reminder
Stephen Kelly
@stevekm

@xg203

my pipeline had performance issue. The java process ate 171 threads. login node was frozen. I had to stop it so other people could use terminal.

This can happen with Nextflow since I believe it spawns a lot of threads both to monitor tasks and to copy files to publishdir. I think I used to use this to keep the number of file copy threads down;

        // number of Nextflow process threads for moving files; should match Makefile 'submit' threads
        filePorter.maxThreads = executor.$local.cpus * 3

where executor.$local.cpus = 8 or some smaller value that I passed in via CLI params

it can also help to limit your executor.queueSize, I think

Stephen Kelly
@stevekm

@pditommaso we are having a really tough time troubleshooting some weird LSF behaviour mskcc/tempo#736, where jobs are getting killed but returning 0 exit codes resulting in broken files to propagate through the pipeline. For some reason in these cases, we keep seeing reports of a USR2 signal being sent (from LSF?) on hitting walltime or mem limits, thus shutting down GATK4 with a 0 exit code instead of a failure code. However, as per the discussion there, I am not clear if Nextflow is handling the USR2 signal in a way that might be contributing to the issue. For example .command.run has:

on_term() {
    set +e
}
trap on_term TERM INT USR1 USR2

I was thinking to try and start setting my own bash traps for USR2 within the Nextflow process script block to try can catch these weird signals and force a non-zero exit code through, but I was afraid that it would override the traps you have set and break things. Do you have any ideas?

Paolo Di Tommaso
@pditommaso
usually it's used to signal soft killing
Kate Murat
@kmurat1
@pditommaso unfortunately same thing happened second process is not accepting demultiplexed files generated in first process it is responding to files from input_ch. Please correct me if I get it wrong as Im generate files and "save" them using into I should be able to extract them using from (or some other operators)?
Paolo Di Tommaso
@pditommaso
from and into does not save or extract data
it's just a way to link tasks
Stephen Kelly
@stevekm
it appears as if after this USR2 signal is being sent, the on_term trap is activating (?) and allowing additional script commands to execute, which were meant to be used to ensure successful completion of the GATK process and raise errors. But either way the tasks are still ending with code 0 in the .exitcode file, even when LSF itself reports an exit code 19 in the logs
Paolo Di Tommaso
@pditommaso
GATK is not very consistent with exit codes
but without a test case hard to say anything
Stephen Kelly
@stevekm
yea the hardest part is that this only happens when you run a pipeline with a lot of jobs, I cannot even reproduce it myself by setting a 1min walltime
a combination of weird signal from LSF + weird behavior from GATK I think
@kmurat1
have you looked at the contents of your paired_samples? Did you try paired_samples.println() or paired_samples.subscribe { println "${it}" } to make sure that the items are coming out in a pair?
Stijn van Dongen
@micans
FWIW @stevekm I'd be surprised if a library needs x permission; @idot mentioned it did work with just one or two input sets
Stephen Kelly
@stevekm

@kmurat1

also this:

set val(name), file(reads) from paired_samples

makes it look like its only accepting a single file as input, if you are outputting an array of files in your previous step and inputting them here then I think it would be like this:

set val(name), file(reads: "*") from paired_samples

but you would have to check to make sure that paired_samples looks something like this:

[ [sample1.r1.fastq, sample1.r2.fastq], [sample2.r1.fastq, sample2.r2.fastq], ... etc ]
xg203
@xg203
@stevekm
@stevekm thank you for your input. should I do the following for thread issue? process BCFtoolsIsec {
cache true
cpus 3
executoe 'lsf'
maxForks 80
filePorter.maxThreads = executor.$local.cpus * 3
….
}
Stephen Kelly
@stevekm

@micans

I'd be surprised if a library needs x permission;

if its not x perms then maybe the task is not executing as the expected user?? Or maybe the lib was installed under a user different account. If this is on HPC you could also have a situation where the job lands on a node where the user profile permissions are not propagating in correctly or not set correctly on that particular node

I have had crazy issues with that kind of stuff before
xg203
@xg203
if that case, I will use 3803 threads. so I need to reduce the maxForks, and change to filePorter.maxThreads = executor.$local.cpus
Stephen Kelly
@stevekm
maxForks only limits the number of concurrent instances of a process, not the total number of process. In that way, it can reduce the total number of tasks being run, but I think that ultimately executor.queueSize is more useful for that. Also you can just supply a value for filePorter.maxThreads, maybe something like 4 or 8 or something like that, see what works
Combiz
@combiz_k_twitter
is it possible to specify the -P exp-XXXXX parameter for PBSPRO jobs? it's for specifying a project account number for billing (express queue)
Combiz
@combiz_k_twitter
can clusterOptions be used in conjunction with vars like queueSize = 100? Presumably one has priority?
Stephen Kelly
@stevekm

@combiz_k_twitter

is it possible to specify the -P exp-XXXXX parameter for PBSPRO jobs? it's for specifying a project account number for billing (express queue)

yeah I am guessing that would probably fall under clusterOptions. But that is independent of queueSize and both can be set at the same time

Combiz
@combiz_k_twitter
thanks, I wonder which has precedence if cpus = 2 memory = '32 GB' is combined with clusterOptions = '-lselect=1:ncpus=6:mem=12Gb -q med-bio -l walltime=24:00:00 -V'
i'll try adding clusterOptions = '-P exp-XXXXX' then and hope it cleverly combines that option with the cpus = x and memory = x defined elsewhere :)
xg203
@xg203
Thanks a lot! @stevekm
twierik
@twierik
Is it possible to output colored text in the terminal such as a warning in red text using println()?
Paolo Di Tommaso
@pditommaso
you need to manage directly ansi codes
twierik
@twierik
Do you have some kind of example? I tried a lot of groovy ansi println suggestions but they all result in errors
it's a mess, I don't advice to do
also println breaks the NF output, use log.info instead
twierik
@twierik
It's a mess for sure... I'll stick to white text haha
but tnx for the log.info tip :)
xg203
@xg203
@stevekm
process BCFtools {
cpus 3
executoe 'lsf'
filePorter.maxThreads executor.$local.cpus * 3
….
}
I got this error : No such variable: filePorter
Stijn van Dongen
@micans
@xg203 try to put this outside a process definition. Not sure if that helps, but it's a logical thing to do.
So filePorter.maxThreads = executor.$local.cpus * 3
xg203
@xg203
@micans

filePorter.maxThreads = executor.$local.cpus * 3

process BCFtools {
cpus 3
executoe 'lsf'
….
}

I got this error : No such variable: executor

Viren Amin
@virenar

I am trying to add multiple container options using containerOptions in nextflow

process get_app_data {
        container "$params.repository/gappd:latest"
        containerOptions "--priviliged"
        containerOptions "--cap-add SYS_ADMIN"
        containerOptions "--device=/dev/fuse"

I am not sure these options are passed when starting the container. Is there any way to debug docker run commands.

Stijn van Dongen
@micans
@xg203 I couldn't get it to work either. Some more information is needed regarding this filePorter.maxThreads -- @stevekm this was perhaps something that worked in the past and no longer, or more context is needed.
twierik
@twierik
I have a process to automatically download a database. This is done by a script which is available in a docker container I'm using . The problem is that docker doesn't allow nextflow to create a new directory within the container to store the downloaded files. Is there some kind of setting I should change to make this possible? If I manually fire up the container I can create directories without problems, but through nextflow this isn't possible for some reason.
Stephen Kelly
@stevekm

@virenar

I am trying to add multiple container options using containerOptions in nextflow

cant you just string them all together in a single directive? Like containerOptions "--priviliged --cap-add SYS_ADMIN --device=/dev/fuse"? Does that not work?

@xg203 @micans the place where I last used the filePorter config was here; https://github.com/NYU-Molecular-Pathology/NGS580-nf/blob/3ba2f970c3fbee56080ba60727f7bf43cb1be3b2/nextflow.config#L344
note that this is inside the nextflow.config file, not in the pipeline.nf file and not inside a process. This is a global setting for the entire pipeline. Also the example shown there is referencing the local executor which I had already set but if you do not have that set explicitly then I would think it would be better to just try something like filePorter.maxThreads = 8 or similar first just to see if it works. I dont have a setup at the moment to test it myself on the latest versions of Nextflow so I am not sure if anything has changed since I wrote that one there in the link
Stijn van Dongen
@micans
@stevekm thanks! something for @xg203 to try.
xg203
@xg203
@micans "filePorter.maxThreads = 8" works. I will monitor threads for large samples later. thanks a lot!
@stevekm thank you!