These are chat archives for nextflow-io/nextflow

29th
Sep 2016
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:40

Hello,
long time...!

The following piece of code works the way I intend (!) but this is not what the docs says (!!):

ch = 'Hello world!'

process go {
    echo true
    storeDir "data/"

    input:
    val f from ch

    output:
    file('output.txt')

    script:
"""
echo $f > output.txt
exit 1
"""
}

The doc about storeDir says:

Whenever a process is successfully completed the files listed in the output declaration block are copied into the directory specified by the storeDir directive.

But clearly data/output.txt is created.
Nextflow 0.22.1

Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:41
ah
hello !
so do I fix the bug of the doc ?
:)
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:46
Well, I just want to make sure you don't fix the code in next release ..
:)
Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:47
I see, but seriously I think this must be considered a bug
why do u need that files if the job failed ?
it could result in a inconsistent state
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:48
I wanted to keep some files for debugging purposes, even though the process fails.
Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:51
ok, maybe by default files are published in case of error but it's possible to add a flag that publish them in any case
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:51
issue in #227
Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:51
great thanks!
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:51
Do your magic for a proper title :)
Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:52
ahah, sure I will give it a magic broken english title
;)
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:54

but it's possible to add a flag that publish them in any case

Huh, future implementation, or does that exist already?

Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:54
no, I mean when fixing the issue I could add that flag
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:58
Sounds good.
Paolo Di Tommaso
@pditommaso
Sep 29 2016 11:58
nice
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 11:59
In the meantime I can also write
echo $f > $SOME_PATH/data/output
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:00
yes, but if it works as you expects just use it, no?
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 12:02
It will break when you fix the bug and I update nextflow's version
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:02
I will notify you ..
anyhow as u prefer, of course
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 12:04
I mean that I 'll forget to fix it, the next time I update nextflow from our production. I have 0.12.0 and 0.17.1 in production right now, and upgrading all pipelines to 0.22.0
I guess next time you'll have 0.30+ :p
But anyway I digress
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:05
I see :)
Hugues Fontenelle
@huguesfontenelle
Sep 29 2016 12:05
tx for looking at this!
Mokok
@Mokok
Sep 29 2016 12:19

Hi ! I was digging in the nextflow sources and i was wondering if it's possible to map an additional executor (ProcessFactory) as an interface. And load an implementation of this interface using a "/lib" dir.
Thus allowing to use a custom external executor provided for each project.

Since groovy scripts are...scripts (not compiled), i guess it could be possible, but as i'm absolutely not used to groovy, i thought you could provide me some wise words. :)

Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:21
um, well groovy scripts are compiled to java classes
the compilation is done on the fly
Mokok
@Mokok
Sep 29 2016 12:22
ok then, i gonna cry in the corner ^^
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:22
:)
apart of this the main problem is the pipeline script uses a different classloader from the nextflow itself
this is the reason the script classloader can be reconfigured dynamically fetching classes and libraries from the lib folder
however you can include custom libraries to nextflow by using the NXF_CLASSPATH env var
Mokok
@Mokok
Sep 29 2016 12:30
well, if i understood, i can include custom libraries to nextflow, and then do what i mentioned above (adding custom executor/taskhandler implementation)
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:30
yep
Mokok
@Mokok
Sep 29 2016 12:30
gonna check this NXF_CLASSPATH out then ! Thanks for your wise words :)
Paolo Di Tommaso
@pditommaso
Sep 29 2016 12:31
it is supposed to be a stable component thus makes sense to deploy it as a lib
welcome