These are chat archives for nextflow-io/nextflow

19th
Sep 2017
Evan Floden
@evanfloden
Sep 19 2017 09:37

https://twitter.com/jeffbarr/status/909830144554016770

So now the full potential of the autoscales terminateWhenIdle can be utilised without worrying about wasted partial cpu time. Correct?

Francesco Strozzi
@fstrozzi
Sep 19 2017 09:41
Yup
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:29
actually I need to update NF code because it's optimised to shutdown instances when reaching hour interval
Evan Floden
@evanfloden
Sep 19 2017 12:30
:+1: I see that min charge is 1 min. I guess the start up time for a NF worker is atleast this?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:32
I think VM bootstrap + NF daemon setup + Docker image download may require 3-4 mins
Evan Floden
@evanfloden
Sep 19 2017 12:33
Good to know
Luca Cozzuto
@lucacozzuto
Sep 19 2017 12:36
Hi guys, it was nice to meet you in person!
I'm wondering Paolo if there is a way to stop NF after running it in background
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:38
sure kill <proc-id>
you can find the process is in the .nextflow.pid file
Luca Cozzuto
@lucacozzuto
Sep 19 2017 12:39
you think having a wrapper to this with a command nextflow stop
or nextflow end <pid>
?
killing processes is a bit ugly :)
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:40
alias 'nextflow stop'=kill
Luca Cozzuto
@lucacozzuto
Sep 19 2017 12:40
eheheh
how you fill when someone kill your tool
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:42
I think we should implement a better management tool, but at this stage killit's a quick workaround
Luca Cozzuto
@lucacozzuto
Sep 19 2017 12:43
+1
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:44
BTW just implemented #413
add a coffee credit in my account ;)
Luca Cozzuto
@lucacozzuto
Sep 19 2017 12:46
we have now 4789104982940 coffees... :) you can open a bar!
Paolo Di Tommaso
@pditommaso
Sep 19 2017 12:47
:)
Francesco Strozzi
@fstrozzi
Sep 19 2017 13:09
a real (napolitan) coffe by the way
:+1:
Paolo Di Tommaso
@pditommaso
Sep 19 2017 13:10
oh, that's out of question
Francesco Strozzi
@fstrozzi
Sep 19 2017 13:23
@pditommaso the point on #422 was not to check if an instance is running using a IAM role, but to let NF start an elastic cluster specifying which is the IAM role the instances should use.
I think it is not possible atm
Paolo Di Tommaso
@pditommaso
Sep 19 2017 13:30
I agree in the goal.
why do you think it should not be possible?
Francesco Strozzi
@fstrozzi
Sep 19 2017 13:35
I haven’t found an option to specify that in NF for the cluster creation….but if I have missed it, well issue resolved :)
Phil Ewels
@ewels
Sep 19 2017 14:31
Hi @pditommaso! I have IntelliJ up and running with nf :+1: However, when I compile, the ./launch.sh script doesn't seem to run the changed version, that only works if I run make compile again. How should I launch the IntelliJ-compiled version for testing?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:32
if you make compile you should make test
if you want to compile with the IDE use build > Rebuild project in the menu
then right click the test case and launch it choosing Run <test name> in the popup menu
or Run <test name> if you want to use break point, inspect vars, etc
Phil Ewels
@ewels
Sep 19 2017 14:37
aha, but this assumes that I want to run a bundled test..? I've been running an actual pipeline each time ;P
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:38
bundled test == unit test, I guess :smile:
so there's a third option
run make compile
then run NF as
./launch -remote-debug run <usual options here>
you will notice the it stop waiting for the debugger to connect to the port printed
then open the IDE
choose Run > Edit configuration ..
(you need to configure the remote debugger)
choose Remote in the left pane
then click the plus icon on the top-left bar
choose Remote in the options list
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:43
give it a name, and above all set the port printed by NF that's 8010
then click OK to confirm
this configuration obviously is required only the very first time
Phil Ewels
@ewels
Sep 19 2017 14:45
ok, then when I run ./launch -remote-debug run something should happen I guess?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:45
finally choose in the main menu Run > Debug ... and select the profile name you have created
it just start and wait for the debugger connection
Phil Ewels
@ewels
Sep 19 2017 14:48
ok, I got something..
Connected to the target VM, address: 'localhost:8010', transport: 'socket'
Disconnected from the target VM, address: 'localhost:8010', transport: 'socket'
(in the Debugger Console)
but not much else..
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:49
it run !
Phil Ewels
@ewels
Sep 19 2017 14:49
yeah it ran!
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:49
:)
Phil Ewels
@ewels
Sep 19 2017 14:49
but I can see an error in .nextflow.log
is that not meant to show up somewhere or something?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:50
no unless you set any breakpoint
open the code you want to check
or you are expecting to be executed
then mouse click in the grey area between the left pane and the editor pane, it should appear a round red-circle
try to re-execute with ./launch.sh -remote-debug run ... it should stop if passing there
Phil Ewels
@ewels
Sep 19 2017 14:53
haha, this is so complicated :laughing:
and I still have to do make compile every time before ./launch?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:53
come on .. ! :D
nope, only if you modify the code
Phil Ewels
@ewels
Sep 19 2017 14:54
..which is every time I want to rerun the test?
haha, anyway, right, trying again with the breakpoint
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:55
well, not always .. sometime you change different condition without modifying the code!
Phil Ewels
@ewels
Sep 19 2017 14:55
aha! ok the debugger thing worked
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:55
this is why python should be forbidden ! :joy:
teach devs to be lazy .. :D
Phil Ewels
@ewels
Sep 19 2017 14:57
haha, yup! I started coding with Flash actionscript ;) Then PHP.. then Perl. Python seems very strict by comparison!
Paolo Di Tommaso
@pditommaso
Sep 19 2017 14:58
impressive career for a flash dev! ;)
Phil Ewels
@ewels
Sep 19 2017 15:00
hah, that bit didn't last for long..
anyway, thanks for the step by step instructions!
v. handy being able to see the debug info
Paolo Di Tommaso
@pditommaso
Sep 19 2017 15:01
yep, still don't understand how people can code w/o an IDE
except for NF code obviously :D
fmorency @fmorency using vim :(
Paolo Di Tommaso
@pditommaso
Sep 19 2017 15:02
anyone has its own perversion ..
Francesco Strozzi
@fstrozzi
Sep 19 2017 15:30
IDEs are terrible
:)
Paolo Di Tommaso
@pditommaso
Sep 19 2017 15:31
;)
Francesco Strozzi
@fstrozzi
Sep 19 2017 15:37
in the end I think “whatever works” it’s my mantra, everyone has it’s own mindset and way of working when coding. For me VIM + a bunch of decent plugins + tmux it’s the best setup. Tried many times using IDEs and it feels like trying to eat spaghetti with a spoon, it’s possible in theory but just doesn’t work for me :)
Venkat Malladi
@vsmalladi
Sep 19 2017 15:38
:smile:
Phil Ewels
@ewels
Sep 19 2017 15:39
I must admit that for me, this feels like I'm trying to crack an egg using a sledgehammer.. ;)
Paolo Di Tommaso
@pditommaso
Sep 19 2017 15:40
LOL
Venkat Malladi
@vsmalladi
Sep 19 2017 16:28
Is there a nextflow styleguide for the .nf files?
Paolo Di Tommaso
@pditommaso
Sep 19 2017 16:30
nope ..
Venkat Malladi
@vsmalladi
Sep 19 2017 16:30
okay
Phil Ewels
@ewels
Sep 19 2017 19:31
Some groups have made their own style guides / templates though
e.g. Mattieu last week presented their template pipeline repository: https://github.com/IARCbioinfo/template-nf
We usually use one of our own previous pipelines as a reference starting place
So I'd recommend having a look through pipelines on the awesome-nextflow list and looking for patterns that make sense to you..
Venkat Malladi
@vsmalladi
Sep 19 2017 19:34
@ewels thanks for the help
trying to establish some kind of standard in our group
Ashley S Doane
@DoaneAS
Sep 19 2017 20:36
I've used the chipseq pipeline from @ewels as a very helpful guide :+1:
related question for genome references in pipelines... I call a python script that takes 12 command line args. I had put these in my nextflow.config using env scope. Now that I have more than one genome, wondering if I can do something like this:
Ashley S Doane
@DoaneAS
Sep 19 2017 20:43
params {
    genomes {
                'hg38'  {
                    bwa = 'hg38.bwa.fasta'
                      env {
                        DNASE_BED =  "dnase_all_p10_ucsc.bed.gz"
                         }
sorry the formatting isnt working...