These are chat archives for nextflow-io/nextflow

8th
Aug 2017
Simone Baffelli
@baffelli
Aug 08 2017 06:22
what, 18 hours??
Simone Baffelli
@baffelli
Aug 08 2017 09:13
Is anyone using groupBy?
I have a question regarding custom sorting closures
Paolo Di Tommaso
@pditommaso
Aug 08 2017 09:16
I would like to deprecate it
Simone Baffelli
@baffelli
Aug 08 2017 09:16
:cry:
that seemed to serve my purpose perfectly
or at least I believed it
Paolo Di Tommaso
@pditommaso
Aug 08 2017 09:17
(sorry meeting)
LukeGoodsell
@LukeGoodsell
Aug 08 2017 10:50
@pditommaso :point_up: August 7, 2017 4:10 PM
When I run a process that has the storeDir directive, the output files are copied to that location. If I re-run the pipeline (without the -resume flag, since I want other processes to re-run) that process is skipped and the output is passed along as normal.
storeDir seems to copy files, as demonstrated here:
https://gist.github.com/LukeGoodsell/06945acf30d35ebd6e2b82bb83e29bec
As you can see in the above output, when the pipeline is run the first time, two copies of the output file are made: one in the work/…/ directory, and one in the storeDir path.
When the command is re-run, the testProc process is skipped and the output file is re-used.
Paolo Di Tommaso
@pditommaso
Aug 08 2017 10:54
when users overcome the author :grin:
LukeGoodsell
@LukeGoodsell
Aug 08 2017 10:55
I’ve tried looking through NF’s source (hoping to make a pull request), but I can’t find where the file is copied
Paolo Di Tommaso
@pditommaso
Aug 08 2017 10:57
file are copied by the .command.run wrapper script
LukeGoodsell
@LukeGoodsell
Aug 08 2017 10:58
I see
Paolo Di Tommaso
@pditommaso
Aug 08 2017 10:58
the that command are written by this code
precisely by this method
actually right now you could specify move to instead of copy that files
would that help ?
LukeGoodsell
@LukeGoodsell
Aug 08 2017 11:00
I think so, though I guess I wouldn’t be able to use -resume afterwards?
(if I wanted to)
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:01
resume would be fine because, it would check the storedir, not the work dir
actually it should be the default . .
LukeGoodsell
@LukeGoodsell
Aug 08 2017 11:03
Cool
Yep, that works perfectly!
Thanks, Paolo :thumbsup:
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:07
:)
NF has too many secrets even for me :)
LukeGoodsell
@LukeGoodsell
Aug 08 2017 11:09
:laughing:
Simone Baffelli
@baffelli
Aug 08 2017 11:24
Paolo, why do you intend to deprecate groupTuple?
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:24
groupBy not groupTuple
Simone Baffelli
@baffelli
Aug 08 2017 11:26
ah
:worried:
i misunderstood you then
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:26
:)
Simone Baffelli
@baffelli
Aug 08 2017 11:26
I thought you wanted to deprecate groupTuple
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:27
nono
Simone Baffelli
@baffelli
Aug 08 2017 11:27
but that one should be exactly the one I need...if only I would understand how to use a custom sorting strategy
Simone Baffelli
@baffelli
Aug 08 2017 11:40
because so far my attempts were quite unsuccessfull
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:44
what' the problem ?
Simone Baffelli
@baffelli
Aug 08 2017 11:47
Suppose I am trying to use groupTuple with a channel emitting Hasmaps, where to each key I have a list of lists
how does groupTuple sort them?
No, wrong. I have lists consisting of [data, timestamp, method] and I want to group them by method.
That works fine, but I cant understand how to make the sort work properly
I always get
No signature of method: _nf_script_a5ffc65e$_run_closure104.call() is applicable for argument types: (sun.nio.fs.UnixPath) values: [path_to_data]
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:52
you can provide a Comparator or a closure that will act as a comparator
see here
Simone Baffelli
@baffelli
Aug 08 2017 11:52
Yes, I'm trying to do thaty
So, my closure should not use the spaceship operator?
Becuase you are comparing in the operator implementation?
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:54
yes, but not sure you can compare paths
convert to strings
Simone Baffelli
@baffelli
Aug 08 2017 11:54
But my problem is that I want to sort lexicographically
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:54
to toString()
Simone Baffelli
@baffelli
Aug 08 2017 11:54
that was another mistake :sweat_smile:
I want to sort by multiple criteria, because actually timestamp is a list of two timestamps
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:55
if so, put sort: true should work
Simone Baffelli
@baffelli
Aug 08 2017 11:55
But will it not also try to sort it by other entries in the list?
I would like to sort only by specified entries, an I thought that would work similarly to toSortedList
Paolo Di Tommaso
@pditommaso
Aug 08 2017 11:56
not sure, cannot check now
Simone Baffelli
@baffelli
Aug 08 2017 11:57
I will then try to hack something together, e.g to convert the two timestamps in a string and compare the strings
Simone Baffelli
@baffelli
Aug 08 2017 12:03
If I understand the source right, I can only access one element at a time and I should return an object that can be compared
Paolo Di Tommaso
@pditommaso
Aug 08 2017 12:52
well, the comparator takes two objects you need to compare
Simone Baffelli
@baffelli
Aug 08 2017 12:58
but I tried and I've got strange results
it returns single elements of the tuple
Paolo Di Tommaso
@pditommaso
Aug 08 2017 12:59
what have you tried ?
Simone Baffelli
@baffelli
Aug 08 2017 13:00
{a, b -> [3] <=> b[3] & a[4] <=> b[4]}
because i want to sort it by both the initial and the final date composing each timestamp, so that at the end I obtain an ordered timeseries
because after grouping, i average each emission, but the averaging should be happening in the right order
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:03
is the above the exact code ?
Simone Baffelli
@baffelli
Aug 08 2017 13:03
yes
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:03
what is [3] ?
Simone Baffelli
@baffelli
Aug 08 2017 13:03
uuups
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:03
and & ?
Simone Baffelli
@baffelli
Aug 08 2017 13:03
:sweat_smile: should be a[3]
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:03
& should be &&
Simone Baffelli
@baffelli
Aug 08 2017 13:04
damn python
using too many languages
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:04
:)
Simone Baffelli
@baffelli
Aug 08 2017 13:22
does groovy have some sort of safe subscripting?
say i want to access a list some_list[1...end] and I receive end from a channel
but I am not always sure that the list is long enough
Paolo Di Tommaso
@pditommaso
Aug 08 2017 13:23
not sure, try it yourself with nextflow console
Simone Baffelli
@baffelli
Aug 08 2017 13:24
I think for the time being i will use remainder: true in groupTuple
to avoid it
it still does not work somehow
Félix C. Morency
@fmorency
Aug 08 2017 13:33
mmm still got an instance where a process has been rerun by -resume and shouldn't have
Félix C. Morency
@fmorency
Aug 08 2017 21:02
aws-batch executor?
Paolo Di Tommaso
@pditommaso
Aug 08 2017 21:03
yes!!
Félix C. Morency
@fmorency
Aug 08 2017 21:03
Nice! Sounds promising! You can go sleep now. :P
Paolo Di Tommaso
@pditommaso
Aug 08 2017 21:04
ahah
there are more important activities before! ;)
Kevin Sayers
@KevinSayers
Aug 08 2017 21:07
:clap:
Paolo Di Tommaso
@pditommaso
Aug 08 2017 21:09
Starry to study aws batch tomorrow @KevinSayers ! :)
*start
Kevin Sayers
@KevinSayers
Aug 08 2017 21:31
:+1: seems quite interesting
Mike Smoot
@mes5k
Aug 08 2017 21:43
:+1: very interested to learn about this!