These are chat archives for nextflow-io/nextflow

12th
Apr 2018
Simone Baffelli
@baffelli
Apr 12 2018 09:25
So metaprogramming?
You want to programmatically name a process?
JezSw
@JezSw
Apr 12 2018 10:37
Have I missed something obvious, we're trying to run some processes natively and some with singularty within the same workflow. Can we use $processname to enable and disable singularity?
JezSw
@JezSw
Apr 12 2018 11:05
This is still coming off the back of our SLURM work. I got slightly excited yesterday and it seems the command @pditommaso game me (below) just sets it to a fixed value at the moment. We're trying to get the one that gets passed to the sbatch script where that works (.command.run)
env.CUDA_VISIBLE_DEVICES='${CUDA_VISIBLE_DEVICES:-1}'
Paolo Di Tommaso
@pditommaso
Apr 12 2018 11:09
Is not that bar supposed to be defined by slurm?
JezSw
@JezSw
Apr 12 2018 11:19
The nodes are supposed to see that variable (and do on non nextflow native and singulaity as well as nextflow without singulairty jobs) to know which GPU to use.
I've been trying to get SINGULARITYENV_<VARIABLE> to pass it through that way too but no luck.
Paolo Di Tommaso
@pditommaso
Apr 12 2018 11:25
I think you should understand is that variable is visible in the . command.run
Eventually remove the :-1 to avoid it default to 1 if not exist
Kevin Sayers
@KevinSayers
Apr 12 2018 11:28
so you are saying --gres still does not appear to be setting the GPU, and the behavior you want is --gres gpu:1 should result in gpu:1 being the GPU used within the Singularity container? Setting CUDA_VISIBLE_DEVICES as the env just overrides it for all processes and you cannot then specify a --gres for each process?
Simone Baffelli
@baffelli
Apr 12 2018 12:00
So, I may have messed with nextflows database by manually editing the log file because it could not find the index of a run
I could run the clean command after the edit, but now the working directory is still quite full
I suppose there's not a lot I can do but start anew?
Of course I knew that editing the history manually would be very stupid
https
Oops
Ok :wave:
Simone Baffelli
@baffelli
Apr 12 2018 12:07
Enjoy
arontommi
@arontommi
Apr 12 2018 12:08

Hi(not aimed at Paolo, since he is on vaycay),

lets say i want to output bam and bai files but not save them to publish dir but save all other output files, would this work? :

    publishDir "${params.outdir}/VariantFiltration", mode: 'copy',
        saveAs: {filename ->
                    if (filename.indexOf(".bam")> 0) null
                    else filename
                    }
Francesco Strozzi
@fstrozzi
Apr 12 2018 13:04
Nextflow run named gigantic_panini
is it a famous scientist of the past ? Or was Paolo just hungry when he added that ? :)
Kevin Sayers
@KevinSayers
Apr 12 2018 13:06
@arontommi if I understand correctly a pattern might be a little more concise publishDir './foo', mode: 'copy', pattern: "*.*[!bam,bai]"
Ancient Indian linguist
arontommi
@arontommi
Apr 12 2018 13:11
@KevinSayers this looks nice! somehow spaced over "pattern", will try.
thanks
Francesco Strozzi
@fstrozzi
Apr 12 2018 13:13
@MaxUlysse interesting, the fact it came out with that adjective was a bit suspicious ;)
Maxime Garcia
@MaxUlysse
Apr 12 2018 13:13
You can get some strange/funny combinations
Simone Baffelli
@baffelli
Apr 12 2018 13:20
I thought of the "Figurine Panini"
Stephen Kelly
@stevekm
Apr 12 2018 17:30
@JezSw "Have I missed something obvious, we're trying to run some processes natively and some with singularty within the same workflow. Can we use $processname to enable and disable singularity?" <- did you figure this out? I might have a go at it later, I think I will end up using a combination of 'modules' and Singularity on some of our workflows. But if you do not define a 'container' for a process then wouldn't Singularity not get invoked? You always have beforeScript available as well to use for other environment customization per-process