These are chat archives for nextflow-io/nextflow

30th
Jan 2018
Fredrik Boulund
@boulund
Jan 30 2018 09:16
Good morning. I have a quick question about Channel.filter. Is it possible to send filtered items to a closure, to print a message about what was filtered?
Paolo Di Tommaso
@pditommaso
Jan 30 2018 09:29
yes
having for example
Channel
    .from( 1, 2, 3, 4, 5 )
    .filter { it % 2 == 1 }
you can do
Channel
    .from( 1, 2, 3, 4, 5 )
    .filter { 
      def filtered=it % 2 == 1
      println "item=$it filtered=$filter"
      return filtered 
   }
Fredrik Boulund
@boulund
Jan 30 2018 09:48
Sweet! Thanks a bunch!
Paolo Di Tommaso
@pditommaso
Jan 30 2018 09:48
:+1:
chdem
@chdem
Jan 30 2018 13:10
Good afternoon
I'm encountering "Unknown config attribute" errors with the 0.27.2 version when I'm trying to access to "workflow.start" or "workflow.runName" or "workflow.sessionId" metadata...
from the config file
everything was ok with the 0.26 version
is it a bug ?
is there another way to access to these metadata ?
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:15
println "$workflow.runName"
println "$workflow.sessionId"
println "$workflow.start"
works just fine
Launching `t.nf` [tender_marconi] - revision: 21eeede4e7
tender_marconi
90accab5-0ef8-4086-a5af-446bdad84885
Tue Jan 30 14:14:48 CET 2018
chdem
@chdem
Jan 30 2018 13:15
ok
from my nextflow.config
    date_folder = $workflow.start
    script_id   = $workflow.runName
    session_id  = $workflow.sessionId
    launch_dir  = $workflow.launchDir
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:17
you need $ only when interpolating that variable in a string ..
like bash
chdem
@chdem
Jan 30 2018 13:18
ERROR ~ Unknown config attribute: start -- check config file: nextflow.config
ok, but same error without $
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:18
and yes, it's not available in the config file that attribute
chdem
@chdem
Jan 30 2018 13:18
ok...
it's not available anymore because I've update nextflow today and this is a new error :worried:
not a problem, I have to put this in my nextflow file
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:20
it don't it think it ever worked in the config file
wait you are right, with version 0.26.x it was working ..
chdem
@chdem
Jan 30 2018 13:23
not a big problem...just have to know that it is not a bug.... :smile:
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:24
nearly
basically with version prior 0.27 it wasn't reported any error
but it won't access that values, just a null
chdem
@chdem
Jan 30 2018 13:26
understood
I've got errors with my script after updating from 0.26 to 0.27.2 and I'm trying to correct that... :smile:
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:28
:+1:
chdem
@chdem
Jan 30 2018 13:30
this is a new error :
Caused by:
  Cannot invoke method replace() on null object
I don't use replace() method
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:31
check the complete error stack trace in the .nextflow.log file
chdem
@chdem
Jan 30 2018 13:32
ok
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:32
copy it here if you need help
chdem
@chdem
Jan 30 2018 13:33
java.lang.NullPointerException: Cannot invoke method replace() on null object
        at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
        at nextflow.processor.TaskProcessor$_bashEnvironmentScript_closure34.doCall(TaskProcessor.groovy:1806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at groovy.lang.Closure.call(Closure.java:414)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForMapEntry(DefaultGroovyMethods.java:5276)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2117)
        at org.codehaus.groovy.runtime.dgm$164.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at nextflow.processor.TaskProcessor.bashEnvironmentScript(TaskProcessor.groovy:1804)
        at nextflow.executor.SimpleFileCopyStrategy.getEnvScript(SimpleFileCopyStrategy.groovy:354)
        at nextflow.executor.ScriptFileCopyStrategy$getEnvScript$3.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
        at nextflow.executor.BashWrapperBuilder.createTaskEnvironment(BashWrapperBuilder.groovy:507)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
        at nextflow.executor.BashWrapperBuilder.build(BashWrapperBuilder.groovy:436)
        at nextflow.executor.BashWrapperBuilder$build.call(Unknown Source)
        at org.codehaus.
my process using the ' \' in the script section to create a multiline command
is it a problem with that ?
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:35
wait
there's something odd with the env variable definitions
chdem
@chdem
Jan 30 2018 13:37
ok
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:37
do you any env var defined in the config file in a dynamic manner ?
chdem
@chdem
Jan 30 2018 13:38
without the '\' I'm throwing another error related with a variable that I'm declaring in the singularity container
Caused by:
  Unknown variable 'GATK_HOME' -- Make sure you didn't misspell it or define somewhere in the script before use it
this is my process that cause the replace() error :
process CallableLoci {

        tag "${prefix}"
        publishDir "${results_outdir}/CALLABLE_LOCI", mode: 'copy'

        input:
        set prefix, file (bam_file) from bams1

        output:
        file("${prefix_output_filename}_${prefix}_CallableLoci.bed")
        file("${prefix_output_filename}_${prefix}.CallableLoci_summary")

        script:
        """
        java -Xmx${task.memory.toGiga()}g \
           -jar \$GATK_HOME \
           -T CallableLoci \
           -R ${fasta_ref} \
           -o ${prefix_output_filename+"_"+prefix}_CallableLoci.bed \
           -minDepth 30 -mmq 1 \
           -L ${intervals_file} \
           -I ${bam_file} \
        -summary ${prefix_output_filename+"_"+prefix}.CallableLoci_summary
        """
    }
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:42
is there anything in the config file that set a env variable ?
chdem
@chdem
Jan 30 2018 13:43
my apologies, sorry to be such a mess, same error replace() with :
script:
        """
        java -Xmx${task.memory.toGiga()}g -jar \$GATK_HOME -T CallableLoci -R ${fasta_ref} -o ${prefix_output_filename+"_"+prefix}_CallableLoci.bed -minDepth 30 -mmq 1 -L ${intervals_file} -I ${bam_file} -summary ${prefix_output_filename+"_"+prefix}.CallableLoci_summary
        """
let me see in the .config file
OK, there is a env.NXF_WORK = in the .config file
I don't know why
everything's correct now....
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:45
one thing at time
chdem
@chdem
Jan 30 2018 13:45
thank you !
now it is OK !!!
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:45
cool
chdem
@chdem
Jan 30 2018 13:46
pffff, I'm sorry
I don't know why, I'm going to investigate in the git history
thank you so much Paolo, you rocks !!
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:46
no pb, I will add an extra check to show a more friendly error message
:)
chdem
@chdem
Jan 30 2018 13:47
Glad it could be useful :smile:
Paolo Di Tommaso
@pditommaso
Jan 30 2018 13:47
indeed !
Thomas Zichner
@zichner
Jan 30 2018 14:01
Hi Paolo,
I have a quick question: I am using publishDir with mode: 'copy' to export some results from the working directory. One of the files to copy is a symbolic link. However, instead of copying the link itself, its target gets copied (which is a few GB in size).
Is there a possibility to avoid that Nextflow follows symbolic links in that case?
Thanks!
Paolo Di Tommaso
@pditommaso
Jan 30 2018 14:03
interesting, I think this enters in the bug category :/
If so, no. You need to resolve the link on your own
could you please open an issue on GitHub for that ?
Thomas Zichner
@zichner
Jan 30 2018 14:05
OK. Good to know. Was not sure if this is a desired behavior. I will open an issue for that.
Thanks
Paolo Di Tommaso
@pditommaso
Jan 30 2018 14:06
doesn't make much sense in this way, if somebody wants the link would use the default
Thomas Zichner
@zichner
Jan 30 2018 14:22
I opened an issue on GitHub: #592
Paolo Di Tommaso
@pditommaso
Jan 30 2018 14:22
very good, thank you
Thomas Zichner
@zichner
Jan 30 2018 14:23
You are welcome. Thanks for all the work you put into Nextflow!
Paolo Di Tommaso
@pditommaso
Jan 30 2018 14:23
it's a pleasure ;)
Fredrik Boulund
@boulund
Jan 30 2018 15:22
@pditommaso Thanks a lot for your help earlier with the filtering of channel contents. It lead to a very nice solution to the problem I've been thinking about for a few weeks now (but haven't gotten time to sit down and actually test, until today).
I have another completely unrelated question
Am I understanding things right that in order to run a workflow directly from a github repository, the mainfile needs to be called main.nf AND the config file needs to be called nextflow.config?
The text in the documentation gives me the impression that I can call the main script file anything, and just define it in nextflow.config using the manifest scope, but that doesn't seem to work, so I guess only my first interpretation is correct.
Maxime Garcia
@MaxUlysse
Jan 30 2018 15:31
otherwise, you can try to do nextflow run organisation/nf-pipeline/script.nf
But I surprized that the manifest declaration did not work
Fredrik Boulund
@boulund
Jan 30 2018 15:34
@MaxUlysse Me too! The way I interpret the documentation text, it sounds very clear that it should be enough to only have the main script declared in nextflow.config as mainScript in the manifest scope.
I'm theoretically fine with being forced to call it main.nf, but I think it's is nicer to be able to call it something else and just declare it in nextflow.config. Not a big deal, however :)
Paolo Di Tommaso
@pditommaso
Jan 30 2018 15:41
yes, you can use any name provided you specify it in the manifest using the mainScript attribute
Fredrik Boulund
@boulund
Jan 30 2018 17:35
Indeed you can! It seems I wasn't paying attention and hadn't merged my most recent changes to the master branch of the repo. :) :palm_tree: (is that a good way of illustrating a face-palm? :D)