These are chat archives for nextflow-io/nextflow

31st
May 2018
Maxime Vallée
@valleem
May 31 2018 06:41
Hello! I am trying to build a config file with different profiles for different clusters. I am scratching my head on a little thing : I would like in both profile to set up, for a process, the java exec path. One cluster has modules and the other has a full path. In the Nextflow script I intend to just have
$javaEXE -jar blabla.jar
but I am not sure if it is the correct way, in the config file :
process {
            withName: HaplotypeCaller {
                beforeScript = 'module load java/openjdk/1.8.0; javaEXE=$(ls $JAVA_HOME/bin/java); echo $javaEXE'
            }
or better : I am sure it is not the correct way because it does not carry :
  .command.sh: line 2: javaEXE: unbound variable
Maxime Garcia
@MaxUlysse
May 31 2018 06:43
try to escape the $ for javaEXE
echo \$javaEXE
Maxime Vallée
@valleem
May 31 2018 06:44
(sorry I let the echo it was for debug purpose and at this step it is holding the correct path, as seen in the log)
in the NF, it is escaped
    script:
    """
    \$javaEXE -XX:ParallelGCThreads=4 -XX:CICompilerCount=4 -Xmx4G -jar ${GATK} \
        HaplotypeCaller \
        -R ${ref} \
        -I ${bam} \
        -O ${bamID}.g.vcf \
        $interval_list_arg \
        -contamination 0 \
        -ERC GVCF        
    """
Maxime Garcia
@MaxUlysse
May 31 2018 06:46
ok
Radoslaw Suchecki
@bioinforad_twitter
May 31 2018 06:46
I think for the first profile, instead of beforeScript..., you need module = 'java/openjdk/1.8.0'
and in the second profile add java to PATH perhaps via beforeScript use env, then no need for $javaEXE in either case
Paolo Di Tommaso
@pditommaso
May 31 2018 06:46
while not using module and proper env variable definition?
Maxime Vallée
@valleem
May 31 2018 06:47
I wanted the flexibility to change the java exec for each process if needed
Maxime Garcia
@MaxUlysse
May 31 2018 06:49
Why not setting it up as a variable in the script statement, but before the actual script?
Maxime Vallée
@valleem
May 31 2018 06:50
I would prefer to have it in the nextflow.config file than in the nf script, if it is what you suggest
Paolo Di Tommaso
@pditommaso
May 31 2018 06:50
best thing would be to handle the java version with its own module at well
Maxime Vallée
@valleem
May 31 2018 06:51
you mean on the second cluster with no modules?
Paolo Di Tommaso
@pditommaso
May 31 2018 06:52
I'm missing something, you are already using module to configure the Java runtime
why do you need that custom variable?
you could just use java -XX:ParallelGCThreads=4 .. etc
Maxime Vallée
@valleem
May 31 2018 06:55
I would like to have 1 NF script, that should run nicely on two clusters, one with modules (and java exec is here $JAVA_HOME/bin/java) and the other without module, but with a full path to it /app/openjdk/.../bin/java
Paolo Di Tommaso
@pditommaso
May 31 2018 06:56
if so, I guess module sets the JAVA_HOME variable
Maxime Vallée
@valleem
May 31 2018 06:57
yes
Paolo Di Tommaso
@pditommaso
May 31 2018 06:57
therefore in your script use $JAVA_HOME/bin/java -XX:ParallelGCThreads=4 ...
and in the config profile that does not use module set a env variable JAVA_HOME for the java path in that cluster
Maxime Vallée
@valleem
May 31 2018 06:58
ha I see
it is clever
Paolo Di Tommaso
@pditommaso
May 31 2018 06:58
Inversion of control applied :wink:
Maxime Vallée
@valleem
May 31 2018 06:59
haha exactly
thank you Paolo for this great tool by the way
Paolo Di Tommaso
@pditommaso
May 31 2018 07:00
you are welcome
Bioninbo
@Bioninbo
May 31 2018 09:40
Hello everyone.
I know how to regroup variables in an array in a channel, like this: test_channel.map { [ it[0,1] , it[2..5] ] }
But I don't know how to ungroup them. i.e. to go back to it[0,1,2,3,4,5] instead of it[0,1,2]. Is it possible?
Paolo Di Tommaso
@pditommaso
May 31 2018 09:54
try it.flatten()
where it is supposed to be an array/list object
Bioninbo
@Bioninbo
May 31 2018 09:58
yes that works. Thanks @pditommaso !
Paolo Di Tommaso
@pditommaso
May 31 2018 09:58
:+1:
Bioninbo
@Bioninbo
May 31 2018 11:10
Another related question: is it possible to select until the last element of the current channel values? I tried this: .map { [ it[0], it[1..it.size()] ] } and .map { [ it[0], it[1..] } but both fail.
Maxime Garcia
@MaxUlysse
May 31 2018 13:07
have you tried -1 ?
Bioninbo
@Bioninbo
May 31 2018 14:01
Yes that works. Thanks @MaxUlysse !
Luca Cozzuto
@lucacozzuto
May 31 2018 15:23
Dear all, I have some doubt with this
process getReadLength {
    input:
    file(single_read_pairs) from read_files_for_size.first()

    output:
    stdout into read_length_for_trimming, read_length_for_index, read_length_for_profile

    script:
    QC.getReadSize(single_read_pairs)
}
it works when stdout into read_length_for_trimming
but not when I add more channels (like the three ones I post before)
@pditommaso is this normal?
Paolo Di Tommaso
@pditommaso
May 31 2018 15:25
code + error = answer
Francesco Strozzi
@fstrozzi
May 31 2018 17:13
you should put that on a T-shirt
Paolo Di Tommaso
@pditommaso
May 31 2018 17:37
:joy:
Jemma Nelson
@fwip
May 31 2018 18:14
I have a question about the storeDir directive. Is it 'safe' to use the same storeDir in separate concurrent nextflow processes? What about separate instances of nextflow, running simultaneously?
Use case is basically exactly the example in the docs: https://www.nextflow.io/docs/latest/process.html?highlight=storedir#storedir - I just want to confirm that there aren't any race conditions with say, running 10 instances of that nextflow script at the same time.
Paolo Di Tommaso
@pditommaso
May 31 2018 19:37
No, that should be avoided
Jemma Nelson
@fwip
May 31 2018 19:40
Okay, thank you :)
Ryo Kita
@rkita
May 31 2018 23:21
Is there a way to prevent the work directory from being created? I've tried playing around with scratch and cache with no luck.
Clarification: I'm ok with it being created, but I want it to be in another location.
Mike Smoot
@mes5k
May 31 2018 23:38
@rkita nextflow run -w /your/work/dir should put you workdir in /your/work/dir
Ryo Kita
@rkita
May 31 2018 23:47
Awesome. Thanks!