These are chat archives for nextflow-io/nextflow

1st
Aug 2018
Timur Shtatland
@tshtatland
Aug 01 2018 03:11
I was interested rather in adding this to all .command.sh files by default. I would prefer not to repeat the same "set -o pipefail" in each script section. This setting is related to set -u and set -e and often used together with these.
Maxime Garcia
@MaxUlysse
Aug 01 2018 07:33
@fstrozzi @apeltzer Did not sort out our new AWS thingy, so haven't been able to try it out yet
sorry
Alexander Peltzer
@apeltzer
Aug 01 2018 08:01
I'll continue today @pditommaso @fstrozzi and will keep everyone posted here and/or open an issue if I find something
Maxime Garcia
@MaxUlysse
Aug 01 2018 08:03
\o/
Francesco Strozzi
@fstrozzi
Aug 01 2018 08:07
@apeltzer :+1:
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:09
does anyone have a hint on remote debugging nextflow? I set NXF_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,address=8000' then connect through ssh tunneling. After getting connection, nextflow terminates with bash: line 0: exec: Listening: not found on the remote machine.
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:18
...same behaviour when I run local and try to attach with jdb -attach localhost:8000... on a nextflow process started with the above mentioned JVM parameters.
Paolo Di Tommaso
@pditommaso
Aug 01 2018 08:25
@lorenzgerber I think I know :smile:
just use launch.sh -remote-debug run .. etc
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:26
aw... hahah, there goes an afternoon playing around ;) Will try... thanks
Paolo Di Tommaso
@pditommaso
Aug 01 2018 08:26
good morning ! :wink:
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:30
@pditommaso We have on a cluster strange behaviour with the pbs executor. Now I set up an ec2 instance with pbspro scheduler to step through it. There seems to be quite some differences between the various pbs/torque versions/implementations...
Paolo Di Tommaso
@pditommaso
Aug 01 2018 08:32
it smell like a new executor contribution :D
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:33
hehe... let's see ;)
Paolo Di Tommaso
@pditommaso
Aug 01 2018 08:36
kidding apart what are the differences ?
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:36
@pditommaso we see sometimes with the pbs executor that all scheduled nextflow jobs have finished but nextflow does not terminate. I suspect some parsing and processing of the qstat output...
Paolo Di Tommaso
@pditommaso
Aug 01 2018 08:37
compare it with the one in the test
Lorenz Gerber
@lorenzgerber
Aug 01 2018 08:44
in the man page I get the following:
  • B Array job has at least one subjob running
  • E Job is exiting after having run
  • F Job is finished
  • H Job is held
  • M Job was moved to another server
  • Q Job is queued
  • R Job is running
  • S Job is suspended
  • T Job is being moved to new location
  • U Cycle-harvesting job is suspended due to keyboard activity
  • W Job is waiting for its submitter-assigned start time to be reached
  • X Subjob has completed execution or has been deleted
so the obvious difference is C
ah, and another difference that I remebmer was the qstat -f -1 you use. I think that does not exist in pbspro...
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:20
@pditommaso local with jdb I get the debugging started, but through an ssh tunnel with intellij, it connects, but then nothing more happens and the run button is greyed out. Any idea?
Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:22
does it work locally, I mean w/o ssh tunneling ?
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:23
yeah, in the remote machine. I can attach with jdb -attach localhost:8010 then I have to start it with 'run'
i mean, i ssh into the remote machine and attach there locally
but I want to attach from my local machine via ssh tunnel to the remote machine obviously. In intellij i get the 'connected' message, but then nothing more happens.
Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:25
have you manage to debug it running NF and intellij in the same machine ?
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:26
yes

i tried now from local with jdb and get the following errors output:
`
jdb -attach localhost:3333
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...

VM Started: No frames on the current call stack
main[1]
`

Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:27
does the remote NF execution proceed when it connect to intellij or it hangs ?
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:27
hangs
Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:27
weird, never used to debug it remotely
do you need that for the qstat stuff ?
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:28
ya, I setup a machine with pbspro on an ec2 because i didn't wanted to mess up my work machine.
but now when i can compile from command line and debug with jdb through an ssh session, it will probably do the trick
Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:30
I usually focus on unit tests and the debug remotely using the logging information
Lorenz Gerber
@lorenzgerber
Aug 01 2018 09:31
yeah, that was my initial idea... then i got carried away by the possiblity to run remote debugging... (ah, i just quickly want to try this also :) hahaha)
Paolo Di Tommaso
@pditommaso
Aug 01 2018 09:31
good luck :smile:
LukeGoodsell
@LukeGoodsell
Aug 01 2018 11:04

Hi there. I’m probably being thick, but is there a way to have a Channel be populated by the output of a closure, à la:

indexedPatientsChannel = Channel.from({ return [1, 2, 3] })

(Obviously I have in mind a more complex closure - this is just for demo purposes)

Paolo Di Tommaso
@pditommaso
Aug 01 2018 11:27
well, it looks this works
process foo {
  input:
  val x from Channel.from({ -> return [1,2,3] })

  """
  echo $x
  """
}
I think -> is not interpreted as a closure
said that I was surprised as well, that the closure is automatically resolved to the actual value . .
LukeGoodsell
@LukeGoodsell
Aug 01 2018 11:32
huh, nifty! Thanks.
LukeGoodsell
@LukeGoodsell
Aug 01 2018 11:40

So it seems val may be implicitly calling .call() against the closure. When map is used against the channel, it just gets a closure object. This works, for example:

myChannel = Channel.from({ -> [1, 2, 3]}.call()).map { println(it); it }

But this does not:

myChannel = Channel.from({ -> [1, 2, 3]}).map { println(it); it }
Paolo Di Tommaso
@pditommaso
Aug 01 2018 11:41
the second should be
myChannel = Channel.from({ -> [1, 2, 3]}).map { println(it.call()); it }
or
myChannel = Channel.from({ -> [1, 2, 3]}).map { println(it()); it }
Phil Ewels
@ewels
Aug 01 2018 13:38
:tada:
Paolo Di Tommaso
@pditommaso
Aug 01 2018 13:45
many !
:tada: :tada: :tada: :tada: :tada: :tada: :tada: :tada:
Francesco Strozzi
@fstrozzi
Aug 01 2018 14:12

```
myList = params.myList?.tokenise(',') ?: []

process foo {
when:
'something' in myList
}
```

@pditommaso I’m getting this error while I’m trying to implement a command line parameter accepting a list of elements

nextflow run test.nf --steps interpro
N E X T F L O W  ~  version 0.31.0
Launching `test.nf` [suspicious_curie] - revision: 5d6d70c9dc
ERROR ~ No signature of method: java.lang.String.tokenise() is applicable for argument types: (java.lang.String) values: [,]
Possible solutions: tokenize(), tokenize(java.lang.Character), tokenize(java.lang.CharSequence), toList(), takeWhile(groovy.lang.Closure)

 -- Check script 'test.nf' at line: 1 or see '.nextflow.log' file for more details
the code is super simple just to test the code
steps = params.steps?.tokenise(',') ?: []
println(steps)
Paolo Di Tommaso
@pditommaso
Aug 01 2018 14:13
Possible solutions: tokenize(),
Francesco Strozzi
@fstrozzi
Aug 01 2018 14:13
jeeez
Paolo Di Tommaso
@pditommaso
Aug 01 2018 14:14
aaaaand ? :smile:
Francesco Strozzi
@fstrozzi
Aug 01 2018 14:14
sorry I completely missed that line
never mind
Paolo Di Tommaso
@pditommaso
Aug 01 2018 14:14
:ok_hand:
Paolo Di Tommaso
@pditommaso
Aug 01 2018 22:02
@tshtatland regarding your question :point_up: July 31, 2018 11:43 PM, yes
you can specify it adding in the nextflow.config a setting like the following
process.shell = ['/bin/bash','-o','pipefail']