These are chat archives for nextflow-io/nextflow

4th
Mar 2019
Tim Dudgeon
@tdudgeon
Mar 04 08:48
@jguhlin Ignite needs a shared file system.
Hugues Fontenelle
@huguesfontenelle
Mar 04 13:42
hi @pditommaso .
You're going to fast regarding #1053 .
@ewels , @MaxUlysse , @drpatelh and myself can all reproduce it.
Please get into an empty dir, and run
nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37
Paolo Di Tommaso
@pditommaso
Mar 04 13:43
I'm sorry or maybe I'm too slow
I'm expecting the exact steps to replicate the issues, I can't reverse engineer your scripts to understand what's wrong
Hugues Fontenelle
@huguesfontenelle
Mar 04 13:45
Perhaps there has been some noise using nextflow config which wasn't that relevant. Or my title sucks ;)
Either way, please run the remote command
not the local one
ie without cloning the code
As for exact steps, I believe that I wrote that at the beginning of the report, ie
# Steps to reproduce the problem
nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37
tough Monday :)
Vladimir Kiselev
@wikiselev
Mar 04 14:53
I am running nextflow with singularity and it put full paths in the singularity exec command. Those paths are not visible inside a container as described here: https://www.nextflow.io/docs/latest/singularity.html#images
What are the settings I can tweak to make it work?
Paolo Di Tommaso
@pditommaso
Mar 04 14:55
or your sysadmins mount that path or you add as an extra (sing) config using runOptions
Vladimir Kiselev
@wikiselev
Mar 04 14:59

looks like:

singularity {
    autoMounts = true
}

fixed this

Thanks @pditommaso
Paolo Di Tommaso
@pditommaso
Mar 04 15:04
:ok_hand:
Luca Cozzuto
@lucacozzuto
Mar 04 16:16
Hi, I see that there is a possibility to increase the required resources in a dynamic way in NF: https://www.nextflow.io/blog/2016/error-recovery-and-automatic-resources-management.html
How can I specify it in the config file?
That should give you a hint - though I'm not entirely 100% sure this is the best way to do it :-)
Luca Cozzuto
@lucacozzuto
Mar 04 16:30
thanks!
Tim Dudgeon
@tdudgeon
Mar 04 17:22
Which executor to use?
I have my nextflow workflows running nicely using the ignite executor, but it seems that this is not really suited for running multiple workflows at the same time.
So it seems I need to look into using a different executor. It would need to be:
  1. free to use
  2. simple to install and configure in a cloud environment
  3. support execution of multiple workflows at once, preferably with different priorities and resource requirements(cores, RAM ...).
  4. support Singularity containers
    Do people here have any particular recommendations? Slurm, Open Grid Engine, PBS, Torque, ....
Alexander Peltzer
@apeltzer
Mar 04 17:37
Slurm works really well here
And wasn’t hard to configure in general
Diogo Silva
@ODiogoSilva
Mar 04 17:50
Hey everyone! I'm trying to retrieve the full configuration of a nextflow pipeline and I've been using nextflow config to that effect, which has been great. However, this seems to take into account only the configuration files and not the main nextflow pipeline file, correct? Is there a way to retrieve the full configuration of a project that takes into account what is in the main pipeline file as well? Thanks
Diogo Silva
@ODiogoSilva
Mar 04 17:58
Oh, are you referring to my question? Is there another more appropriate channel?
Alexander Peltzer
@apeltzer
Mar 04 18:02
ah not my client just showed wrong
Diogo Silva
@ODiogoSilva
Mar 04 18:03
You left me wondering what did I do wrong... :'(
Paolo Di Tommaso
@pditommaso
Mar 04 18:06
configuration is supposed to go in the configuration files
Alexander Peltzer
@apeltzer
Mar 04 18:06
Youre right here diogo
Diogo Silva
@ODiogoSilva
Mar 04 18:12
Ah, yes I completely agree Paolo, but there are several pipelines where parameters are also specified on the .nf file. It's also frequent to see process directives being set within the process. For instance, the label directive seems to be specified only there. And these usually take precedence, right?
Hm, well the label is a bad example, but there's still the other possible directives
Paolo Di Tommaso
@pditommaso
Mar 04 18:14
currently this is not not possible, eventually in the feature future ..
between the other things are you keeping an eye on #984? your feedback would be interesting considering your experience with flowcraft
Diogo Silva
@ODiogoSilva
Mar 04 18:16
Ok, thank you for the information :+1:
Ah, I had seen the tweet but not this issue. Will take a look :D
Joseph Guhlin
@jguhlin
Mar 04 18:30
@tdudgeon Ah cool, thanks
Stephen Kelly
@stevekm
Mar 04 19:34
@pditommaso when running with -resume, is Nextflow supposed to re-copy files from cached processes using publishDir "${params.outputDir}", mode: 'copy', overwrite: true ??
I am getting massive lag when completeing resumed runs because Nextflow is re-copying all the publishDir files over again, even ones that already exist in the publishDir
Paolo Di Tommaso
@pditommaso
Mar 04 19:35
I think so
Stephen Kelly
@stevekm
Mar 04 19:40

I see that in the docs, overwrite = true is actually a default;

https://www.nextflow.io/docs/latest/process.html#publishdir

overwrite When true any existing file in the specified folder will be overridden (default: true during normal pipeline execution and false when pipeline execution is resumed).

it sounds like simply omitting the overwrite: true here might resolve this and prevent re-copying all cached process files. But if I run with -resume, what happens to output from new re-submitted processes? Would the default value of overwrite: false prevent the new files from getting copied to the publishDir because results from a previous run already exist?

Paolo Di Tommaso
@pditommaso
Mar 04 19:42
frankly I don't remember all possible combinations, make a test
Stephen Kelly
@stevekm
Mar 04 20:20
ok cool it looks like it works even better than I thought, I ran through the possible combinations here
so if I simply leave out overwrite: true it should fix the entire issue, I think
Timothy R. Fallon
@photocyte
Mar 04 20:33
Does anyone know an automatic code formatter for nextflow, e.g. something like https://prettier.io ?
Stephen Kelly
@stevekm
Mar 04 20:44
I am not clear exactly what that thing does but I just use the Nextflow language support in Atom
lastwon1216
@lastwon1216
Mar 04 20:44
Hello, is there any way I can get the first part of the attribute before "-" to output files?
For example,
/some/path/file-R1.fastq.gz -> file
Tried .simpleName it would bring out file-R1
Timothy R. Fallon
@photocyte
Mar 04 21:01
@lastwon1216 I think split or tokenize in Groovy should work? See http://www.tothenew.com/blog/groovy-tokenize-vs-split/
lastwon1216
@lastwon1216
Mar 04 22:09
@photocyte thanks for the info!
i used
def (x, y) = rep2-R{1,2}.split('-')
it comes out as [rep2
is there any way to get rid of that [ ?
i tried split and tokenize, but it would not bring just rep2
Timothy R. Fallon
@photocyte
Mar 04 22:14
Not super familiar with groovy, and unclear what the {1,2} is doing exactly, but sounds like the string output from that (likely list-like) object includes the "[". I think you would be able to get the individual item in that list using [0] and [1], and then split on that.
lastwon1216
@lastwon1216
Mar 04 22:22
me neither :( but it seems like it also collects [ from ['rep2', 'R1']
Timothy R. Fallon
@photocyte
Mar 04 22:24
You might consider using regex and grouping to get exactly what you want, and ignore the other characters pretty much. See here: http://mrhaki.blogspot.com/2009/09/groovy-goodness-matchers-for-regular.html
Timothy R. Fallon
@photocyte
Mar 04 23:12
@stevekm I have Atom with Nextflow language support installed. I see the syntax highlighting, but not seeing a way to reformat the code to a particular style. Am I missing something?
lastwon1216
@lastwon1216
Mar 04 23:50
@photocyte i see. I will try that. thank you!