These are chat archives for nextflow-io/nextflow

29th
Jun 2017
nik-sm
@nik-sm
Jun 29 2017 01:05
I'd like to upload outputs from a process to s3. Is there a recommended/"best" way to do this?
(e.g. aws s3 cp vs publishDir?)
Mike Smoot
@mes5k
Jun 29 2017 03:23
publishDir fine for this
Paolo Di Tommaso
@pditommaso
Jun 29 2017 07:39
@nik-sm exactly, you can do something like the following
process foo {
  publishDir 's3://your-bucket/path'
  '''
  your_command --here
 '''
}
Simone Baffelli
@baffelli
Jun 29 2017 08:51

Hello. Is it currently possible to define "custom" channel operators as a sequence of normal operators? I was considering to open an issue because it would find it very useful. I notice that I tend to repeat the same sequence of operators in several locations in my script. Things like:

source_channel.
.combine(other_channel)
.toSortedList()
.map{it->it.transpose()}

It would be very convenient if I could define these operations as a function only once at the begnning of my script (or even in a separate groovy class). The only current solution is to directly add methods to "DataflowExtensions" i guess.

Paolo Di Tommaso
@pditommaso
Jun 29 2017 08:52
at this time you can create a method with the logic eg
def custom_op(source_channel) {
  source_channel. 
  .combine(other_channel)
  .toSortedList()
  .map{it->it.transpose()}
}
tho you will need to invoke as a function not an operator
said that, yes I have some ideas to have an extensible operator mechanism
feel free to open a feature request for that
Simone Baffelli
@baffelli
Jun 29 2017 09:11
Nice workaround :) I guess to call it I need to use the map operator?
Paolo Di Tommaso
@pditommaso
Jun 29 2017 09:14
nope, like other function eg
custom_op(some_other_channel)
obviously the method returns a new channel that can be used as usual
Simone Baffelli
@baffelli
Jun 29 2017 09:20
I see
nik-sm
@nik-sm
Jun 29 2017 13:44
@pditommaso @mes5k Ok, thanks for the reply and the example!
amacbride
@amacbride
Jun 29 2017 23:26
Howdy @pditommaso , congrats on getting 0.25.1 out. I did notice one thing that's more of a nice-to-have: when a foreign-file copy process starts (and mine tend to be long-lived), I'm seeing this warning, presumably because the TaskPollingMonitor is only looking at user-defined tasks and not implicit/internal Nextflow ones:
Jun-29 19:31:07.524 [Actor Thread 17] DEBUG nextflow.processor.TaskProcessor - Copying to process workdir foreign file: s3:///some-bucket-name/example.fastq.gz
Jun-29 19:35:52.350 [Task monitor] WARN  n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled (see the log file for details)
It all eventually works fine, but it gave me a heart attack when I first saw it. Perhaps make foreign-file copy tasks visible to the TaskPollingMonitor?