@idot
File "/usr/lib64/python3.7/tempfile.py", line 258, in _mkstemp_inner fd = _os.open(file, flags, 0o600) PermissionError: [Errno 13] Permission denied: '/dev/shm/pym-29180-mpuki4mm'
obv it seems that the permissions on this file are set to 600, which I belive is user read/write only, but it appears to be a library of some kind and so it might need 700 permissions e.g. read/write/execute. Also, its in your /dev/shm shared memory, was that intentional? Either way you should check on how this file is getting to be loaded in this location and prob do a chmod u+x on it
@xg203
my pipeline had performance issue. The java process ate 171 threads. login node was frozen. I had to stop it so other people could use terminal.
This can happen with Nextflow since I believe it spawns a lot of threads both to monitor tasks and to copy files to publishdir. I think I used to use this to keep the number of file copy threads down;
// number of Nextflow process threads for moving files; should match Makefile 'submit' threads
filePorter.maxThreads = executor.$local.cpus * 3where executor.$local.cpus = 8 or some smaller value that I passed in via CLI params
it can also help to limit your executor.queueSize, I think
@pditommaso we are having a really tough time troubleshooting some weird LSF behaviour mskcc/tempo#736, where jobs are getting killed but returning 0 exit codes resulting in broken files to propagate through the pipeline. For some reason in these cases, we keep seeing reports of a USR2 signal being sent (from LSF?) on hitting walltime or mem limits, thus shutting down GATK4 with a 0 exit code instead of a failure code. However, as per the discussion there, I am not clear if Nextflow is handling the USR2 signal in a way that might be contributing to the issue. For example .command.run has:
on_term() {
set +e
}
trap on_term TERM INT USR1 USR2I was thinking to try and start setting my own bash traps for USR2 within the Nextflow process script block to try can catch these weird signals and force a non-zero exit code through, but I was afraid that it would override the traps you have set and break things. Do you have any ideas?
USR2 signal is being sent, the on_term trap is activating (?) and allowing additional script commands to execute, which were meant to be used to ensure successful completion of the GATK process and raise errors. But either way the tasks are still ending with code 0 in the .exitcode file, even when LSF itself reports an exit code 19 in the logs
have you looked at the contents of your
paired_samples? Did you try paired_samples.println() or paired_samples.subscribe { println "${it}" } to make sure that the items are coming out in a pair?
@kmurat1
also this:
set val(name), file(reads) from paired_samples
makes it look like its only accepting a single file as input, if you are outputting an array of files in your previous step and inputting them here then I think it would be like this:
set val(name), file(reads: "*") from paired_samples
but you would have to check to make sure that paired_samples looks something like this:
[ [sample1.r1.fastq, sample1.r2.fastq], [sample2.r1.fastq, sample2.r2.fastq], ... etc ]
cache true
cpus 3
executoe 'lsf'
maxForks 80
filePorter.maxThreads = executor.$local.cpus * 3
….
}
@micans
I'd be surprised if a library needs x permission;
if its not x perms then maybe the task is not executing as the expected user?? Or maybe the lib was installed under a user different account. If this is on HPC you could also have a situation where the job lands on a node where the user profile permissions are not propagating in correctly or not set correctly on that particular node
maxForks only limits the number of concurrent instances of a process, not the total number of process. In that way, it can reduce the total number of tasks being run, but I think that ultimately executor.queueSize is more useful for that. Also you can just supply a value for filePorter.maxThreads, maybe something like 4 or 8 or something like that, see what works
@combiz_k_twitter
is it possible to specify the
-P exp-XXXXXparameter for PBSPRO jobs? it's for specifying a project account number for billing (express queue)
yeah I am guessing that would probably fall under clusterOptions. But that is independent of queueSize and both can be set at the same time
clusterOptions = '-P exp-XXXXX' then and hope it cleverly combines that option with the cpus = x and memory = x defined elsewhere :)
println breaks the NF output, use log.info instead
process BCFtools {
cpus 3
executoe 'lsf'
filePorter.maxThreads executor.$local.cpus * 3
….
}filePorter.maxThreads = executor.$local.cpus * 3
filePorter.maxThreads = executor.$local.cpus * 3
process BCFtools {
cpus 3
executoe 'lsf'
….
}I got this error : No such variable: executor
I am trying to add multiple container options using containerOptions in nextflow
process get_app_data {
container "$params.repository/gappd:latest"
containerOptions "--priviliged"
containerOptions "--cap-add SYS_ADMIN"
containerOptions "--device=/dev/fuse"I am not sure these options are passed when starting the container. Is there any way to debug docker run commands.
filePorter config was here; https://github.com/NYU-Molecular-Pathology/NGS580-nf/blob/3ba2f970c3fbee56080ba60727f7bf43cb1be3b2/nextflow.config#L344note that this is inside the nextflow.config file, not in the pipeline.nf file and not inside a process. This is a global setting for the entire pipeline. Also the example shown there is referencing the local executor which I had already set but if you do not have that set explicitly then I would think it would be better to just try something like
filePorter.maxThreads = 8 or similar first just to see if it works. I dont have a setup at the moment to test it myself on the latest versions of Nextflow so I am not sure if anything has changed since I wrote that one there in the link