These are chat archives for nextflow-io/nextflow

12th
Jun 2018
Phil Ewels
@ewels
Jun 12 2018 07:15
Morning @pditommaso! Is there a way to determine if a workflow is using docker or singularity? eg. workflow.docker.enabled or something?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:22
there's worklfow.containerEngine and I've just realised that not documented ..
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:22
Some time ago @boulund asked if independent processes could continue if process X fails. The answer was: use process.errorStrategy = 'finish'. This allows already submitted processes to continue. Is there some fundamental reason why nextflow could not continue submitting processes if/until a point is reached when a process is directly or indirectly dependant on input from X?
Phil Ewels
@ewels
Jun 12 2018 07:24
Perfect, thanks @pditommaso!
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:24
:+1:
Is there some fundamental reason why nextflow could not continue submitting processes if/until a point is reached when a process is directly or indirectly dependant on input from X?
what do you mean?
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:27
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:27
not getting the point
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:28
hold o
on
if e.g. fasta2mockFASTQ fails, the whole pipeline will stop, although everything except MOCK_generateReportMatter could be executed
or rather the other paths in the graph do not depend on this process
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:32
if you don't wont to pipeline to stop use process.errorStrategy = 'ignore'
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:34
doh!
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:34
:)
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:35
too obvious I guess :grin:
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:35
exactly
btw, if I not wrong you were going to present NF to a lab meeting or something similar
how was the feedback ?
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:35
correct
It was at C3DIS conference and although the talk was a bit broader, I'd rather just focus on nextflow :smile: - as for feedback, there were a few people who approached me with some more detailed questions
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:40
very nice
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:40
slides here
Later this week I will try to sell the idea to a better targeted audience
People doing bioinformatics and data science, so here the focus would be more on productivity gains
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:43
very nice presentation and thanks for endorsing NF
sounds great
ah there was also Brandon the amazon guy
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:44
Excellent work, pleasure to endorse!
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:44
thanks!
Maxime Garcia
@MaxUlysse
Jun 12 2018 07:46
@bioinforad_twitter Very nice talk
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:50
Thanks @MaxUlysse!
Paolo Di Tommaso
@pditommaso
Jun 12 2018 07:50
don't forget to include the Bioconda support in latest version
:wink:
Radoslaw Suchecki
@bioinforad_twitter
Jun 12 2018 07:51
Yep, was looking forward to give this a go!
Simone Baffelli
@baffelli
Jun 12 2018 10:02
Hello! Is there a way to include all subfolders in bin in the PATH?
I'm trying to do something very ugly
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:03
I'm trying to do something very ugly
:joy:
NF prevents ugly things ..
you can try to use $baseDir/bin/your/ugly/things
Simone Baffelli
@baffelli
Jun 12 2018 10:08
I know i should be using a docker container
but I just want to pack some binaries from the wonderful :worried: PolSARPro software in my project
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:10
docker nor singularity is an option for you ?
Simone Baffelli
@baffelli
Jun 12 2018 10:12
well I just want to test some things quickly
Simone Baffelli
@baffelli
Jun 12 2018 10:23
But I guess its time to learn docker
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:23
:+1:
Simone Baffelli
@baffelli
Jun 12 2018 10:34
Does not seem to hard, but.... can I use a docker container for a process only?
and the rest running on my regular os?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:34
Yes
Simone Baffelli
@baffelli
Jun 12 2018 10:35
so If i specify it only for tha tprocess
the rest will run regularly?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:35
Yes
Simone Baffelli
@baffelli
Jun 12 2018 10:36
that is fantastic :clap:
Paolo Di Tommaso
@pditommaso
Jun 12 2018 10:36
Welcome at the party :smile:
*to
Maxime Garcia
@MaxUlysse
Jun 12 2018 11:26
Hi @pditommaso follow up question on yesterday's file import, how can I use Channel?
By the way, the import static nextflow.Nextflow.file worked very well
I'm looking at the Nextflow code, to understand wich lib I should import for Channel, but more complicated that I had estimated ;-)
Paolo Di Tommaso
@pditommaso
Jun 12 2018 11:48
why?
import nextflow.Nextflow.Channel.*
Maxime Garcia
@MaxUlysse
Jun 12 2018 11:52
Not working
Paolo Di Tommaso
@pditommaso
Jun 12 2018 11:53
what is not working ?
Maxime Garcia
@MaxUlysse
Jun 12 2018 11:53
I got
Apparent variable 'Channel' was found in a static scope but doesn't refer to a local variable, static field or class. Possible causes:
You attempted to reference a variable in the binding or an instance variable from a static context.
You misspelled a classname or statically imported field. Please check the spelling.
You attempted to use a method 'Channel' but left out brackets in a place not allowed by the grammar.
and if I try import static nextflow.Nextflow.Channel.*, I have:
unable to resolve class nextflow.Nextflow.Channel

By the way, there a typo in the error message:

The lookup for myUtils caused a failed compilaton.

But I'm guessing it's groovy

Paolo Di Tommaso
@pditommaso
Jun 12 2018 11:56
sorry my fault, import nextflow.Nextflow.Channel
then use it as Channel.thisAntThat()
Maxime Garcia
@MaxUlysse
Jun 12 2018 11:58
still have the unable to resolve class nextflow.Nextflow.Channel
Paolo Di Tommaso
@pditommaso
Jun 12 2018 11:59
makes no sense, if you were able to import static nextflow.Nextflow.file it should work also this
Maxime Garcia
@MaxUlysse
Jun 12 2018 12:01
I don't have this unable to resolve class when I do import nextflow.Nextflow.Channel.*
Paolo Di Tommaso
@pditommaso
Jun 12 2018 12:12
and ?
Maxime Garcia
@MaxUlysse
Jun 12 2018 12:15
Still not working, with error:
Apparent variable 'Channel' was found in a static scope but doesn't refer to a local variable, static field or class. Possible causes:
You attempted to reference a variable in the binding or an instance variable from a static context.
You misspelled a classname or statically imported field. Please check the spelling.
Edgar
@edgano
Jun 12 2018 12:33
:clap:
Maxime Garcia
@MaxUlysse
Jun 12 2018 13:03
@pditommaso Manage to make it work with import nextflow.Channelactually
Thanks for pointing on the right direction
Paolo Di Tommaso
@pditommaso
Jun 12 2018 13:03
nice, what was the problem ?
Maxime Garcia
@MaxUlysse
Jun 12 2018 13:04
Channel was not in nextflow.Nextflow.Channel, but in nextflow.Channel
Paolo Di Tommaso
@pditommaso
Jun 12 2018 13:05
shit .. :joy:
I'm not able to read my own code
Maxime Garcia
@MaxUlysse
Jun 12 2018 13:07
How many lines of codes?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 13:08
nearly 80k :wink:
Maxime Garcia
@MaxUlysse
Jun 12 2018 13:08
I guess it's a proof that you're human and makes mistakes like the rest of us ;-)
Paolo Di Tommaso
@pditommaso
Jun 12 2018 13:09
ahah
Maxime Garcia
@MaxUlysse
Jun 12 2018 13:09
Next step, trying to prove that you also sleep :-)
Paolo Di Tommaso
@pditommaso
Jun 12 2018 13:09
I should ..
micans
@micans
Jun 12 2018 14:31
Is it possible to run two different github branches from the same nextflow projects at the same time? Looking at $HOME/.nextflow/assets it seems not?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 14:31
no
micans
@micans
Jun 12 2018 14:33
Thanks .. fair enough. I have a little bit of a use case but I can imagine it's not a common one.
micans
@micans
Jun 12 2018 14:46
If one nextflow session is running, is it safe to update the code, pull the new version, and run another session concurrently, as far as config files and nf files is concerned (i.e. they are loaded at start I presume)?
(this would be the same branch btw)
Paolo Di Tommaso
@pditommaso
Jun 12 2018 15:02
If one nextflow session is running, is it safe to update the code, pull the new version, and run another session concurrently,
yes, as long as you don't use any bin scripts
which may be invoked at any time
in time case better better to create a separate local clone on work on that
micans
@micans
Jun 12 2018 15:04
OK, cheers, I did think bin was off-limits.
Paolo Di Tommaso
@pditommaso
Jun 12 2018 15:04
:+1:
Felix Kokocinski
@fkokocinski
Jun 12 2018 16:16
Hi Paolo,
as mentioned on another chat, I'm trying to set up my NF pipeline with an additional config file. This config is required for the programs executed to find input files and direct their output.
To make this Nextflow-compatible I'm now using an approach where I replace the output path with the the NF work path in the config and pass on all files from process to process. Also, stageIn / out mode 'copy' seems to be required.
I'm trying to show this in a simplified test pipeline here: https://bitbucket.org/fkokocinski/nextflow-test/src/master/
This seems to do the right thing, but is this stable in a distributed environment or are there better ways?
Many thanks, Felix
Paolo Di Tommaso
@pditommaso
Jun 12 2018 16:20
in principle there's no need to use stageInMode or stageOutMode to have it running in a distributed environment
why do you need that ?
Felix Kokocinski
@fkokocinski
Jun 12 2018 16:30
Ah, good. I though it might be safer as the next process relies on the output files...
Paolo Di Tommaso
@pditommaso
Jun 12 2018 16:31
as far as the downstream process does not change any *input* file that's fine
NF is based on functional principle, a task should never modify an input file
(in that case you would need a stageInMode='copy')
Felix Kokocinski
@fkokocinski
Jun 12 2018 16:48
Otherwise the approach with pwd as outdir should be OK?
Paolo Di Tommaso
@pditommaso
Jun 12 2018 16:50
what do you mean for the approach with pwd as outdir ?
Felix Kokocinski
@fkokocinski
Jun 12 2018 16:53
Sorry, I mean: replacing the output directory defined in the program's config file with Nextflow's work dir using pwd.
Paolo Di Tommaso
@pditommaso
Jun 12 2018 16:54
well, there you can do whatever you want as long you program is happy with that
there's no magic, that just a bash script NF runs for you
the only constraint is that you need to work in the task directory assigned by NF
Felix Kokocinski
@fkokocinski
Jun 12 2018 20:22
I'll give this a try then. Thanks, Paolo.
Jemma Nelson
@fwip
Jun 12 2018 21:30
I'm doing development of a nextflow pipeline on my local macbook - unfortunately, Docker for Mac is pretty slow at volume mounting. Is there a setting I can use with nextflow to use the 'consistency' bind mount options? https://docs.docker.com/docker-for-mac/osxfs-caching/ (i.e: the :cached in docker run -v /my/mnt/dir:/mnt:cached ...)
Oh, nvm, it's docker.mountFlags. Figures I only find it in the docs after asking ^^;
Felix Kokocinski
@fkokocinski
Jun 12 2018 21:57
@fwip Sounds interesting! Which flags are you setting, please?
Jemma Nelson
@fwip
Jun 12 2018 21:58
Ah, setting docker.mountFlags = 'cached' in my config file speeds up execution of my tests quite a bit
you probably won't see any effect from it unless you're also using docker for mac; it relaxes the file consistency guarantees some and my processes are no longer all disk-bound.
Felix Kokocinski
@fkokocinski
Jun 12 2018 22:39
Thanks, I'll give it a try.