These are chat archives for nextflow-io/nextflow

16th
May 2016
Jason Byars
@jbyars
May 16 2016 19:45
for errorStrategy 'retry' is there any option to do the retries in separate working directories?
Paolo Di Tommaso
@pditommaso
May 16 2016 19:48
It does by default
Jason Byars
@jbyars
May 16 2016 19:55
ok, then it looks like my bugs are far stranger than I originally anticipated. BTW, cfncluster + nextflow appears to mostly work. I think the cluster auto scaling might be triggering some job rescheduling
Paolo Di Tommaso
@pditommaso
May 16 2016 19:57
great, if you could write a blog post describing it, it would great for the community
Jason Byars
@jbyars
May 16 2016 19:57
if I can beat these bugs I will.
Paolo Di Tommaso
@pditommaso
May 16 2016 19:57
Of course, at your convenience!
Jason Byars
@jbyars
May 16 2016 20:02
the one wrench in the works, I'll probably need help on is the 0 initial workers scenario. I.E. just the head node is running and nextflow submits x jobs. It takes a few minutes before new workers come online to service the jobs.
In a normal cluster you would want an error telling you there are no available workers for the submitted jobs. Instead I need some sort of timeout, to say queue the jobs and wait x amount of time. If we still have no workers, report an error.
Paolo Di Tommaso
@pditommaso
May 16 2016 20:06
currently there's not timeout, so jobs just wait for workers to spin up, right?
Jason Byars
@jbyars
May 16 2016 20:29
yes, but nextflow reports an error, scheduling jobs with 0 workers. I'll get an example together for this.
Paolo Di Tommaso
@pditommaso
May 16 2016 20:30
ok, let me know
even better, details this into a feature request. I could manage to include in the upcoming release
Jason Byars
@jbyars
May 16 2016 20:39
I will, I've got to understand the nextflow retry/rescheduling behavior a little better first.
Paolo Di Tommaso
@pditommaso
May 16 2016 20:39
a job rescheduled only if it returns a non-zero error code
Jason Byars
@jbyars
May 16 2016 20:41
what happenes when there is no return? I.E. something died
Paolo Di Tommaso
@pditommaso
May 16 2016 20:41
it's assumed non-zero as well
Jason Byars
@jbyars
May 16 2016 20:44
Great, then this may be more odd SGE + cfncluster trivia than details on nextflow. What I was seeing over the weekend appeared to be jobs rescheduling before the previous run had exited. It was truly bizarre.
Paolo Di Tommaso
@pditommaso
May 16 2016 20:46
weird, who was rescheduling jobs? nxf or sge?
Jason Byars
@jbyars
May 16 2016 20:47
I haven't figured out yet. I just noticed several times there were several more workers and jobs scheduled than there should be. Right now I'm going back and checking the processes for mistakes.
Paolo Di Tommaso
@pditommaso
May 16 2016 20:48
ok
Jason Byars
@jbyars
May 16 2016 20:54
I want to do some contrived super simple test cases, to make useful test cases
Paolo Di Tommaso
@pditommaso
May 16 2016 20:55
makes sense
Jason Byars
@jbyars
May 16 2016 21:03
what are you using for automated testing, so I can create some appropriate cases?
Paolo Di Tommaso
@pditommaso
May 16 2016 21:04
if you don't feel comfortable with it, Junit is supported as well
Jason Byars
@jbyars
May 16 2016 21:22
Spock looks interesting. I'll give it a try
Paolo Di Tommaso
@pditommaso
May 16 2016 21:23
It's very cool, I would strongly suggest
Mike Smoot
@mes5k
May 16 2016 23:36

Hi, I'm wondering if I've uncovered a bug in the expected behavior of the 'distinct' channel operator. With this example I don't see what I expect:

ch = Channel.from([1, 'a'],[2, 'a'], [1, 'a'], [2, 'a'] ,[2, 'a']).distinct().println()

I expect to see just [1, 'a'], [2, 'a'], but I see [1, 'a'], [2, 'a'], [1, 'a'], [2, 'a']. Or have I perhaps misunderstood?

Mike Smoot
@mes5k
May 16 2016 23:45
Aaaand I've mistaken distinct for unique. Sigh.