These are chat archives for nextflow-io/nextflow

20th
Mar 2018
Caspar
@caspargross
Mar 20 2018 10:44

I want to use a result of the .cross() operator as a process input. How can i access the sublists? If the channel is:

[ [1, alpha], [1, x] ]
[ [1, alpha], [1, y] ]
[ [1, alpha], [1, z] ]

i tried to do

set  [id1, name],  [id2, name2] from channel

but this does not work

Paolo Di Tommaso
@pditommaso
Mar 20 2018 10:47
I wish NF had the ability to understand users need and modify the syntax accordingly :)
Caspar
@caspargross
Mar 20 2018 10:48
that would be great :D
Paolo Di Tommaso
@pditommaso
Mar 20 2018 10:49
cross is a bit tricky to use, have you tried combine ?
Caspar
@caspargross
Mar 20 2018 10:51
hm Im not sure if that works, i want to do a join and recycle the elements in one of the channels
Paolo Di Tommaso
@pditommaso
Mar 20 2018 10:53
A second version of the combine operator allows you to combine between them those items that share a common matching key. The index of the key element is specified by using the by parameter (the index is zero-based, multiple indexes can be specified with list a integers).
anyhow if you want to use cross you will need an example map eg
ch.cross().map { a, b -> return [ a[0], a[1], b[1] ] }
Caspar
@caspargross
Mar 20 2018 11:35
thanks!
Luca Cozzuto
@lucacozzuto
Mar 20 2018 12:44
Hi, I have a process that outputs a list of files. I'm wondering how to output them in a channel so that every file is analyzed indipendently
Caspar
@caspargross
Mar 20 2018 12:46
I had this same problem a few days ago. I used the transpose operator
Luca Cozzuto
@lucacozzuto
Mar 20 2018 12:47
in which way?
Caspar
@caspargross
Mar 20 2018 12:48
then just use .transpose() on the next process input. Every file gets used independently in the following process
Luca Cozzuto
@lucacozzuto
Mar 20 2018 12:55
mmm, it does not...
[/nfs/software/bi/biocore_tools/git/nextflow/C4LWG-2018/test0/work/9a/06a120239c0fc3e9d1cbfe6aaac5eb/seq_1, /nfs/software/bi/biocore_tools/git/nextflow/C4LWG-2018/test0/work/9a/06a120239c0fc3e9d1cbfe6aaac5eb/seq_2, /nfs/software/bi/biocore_tools/git/nextflow/C4LWG-2018/test0/work/9a/06a120239c0fc3e9d1cbfe6aaac5eb/seq_3]
this is the output of the first process
Caspar
@caspargross
Mar 20 2018 12:59
I think the difference is that I also used a common ID not in the list for the transposition. Have you tried .flatten() ?
Paolo Di Tommaso
@pditommaso
Mar 20 2018 13:17
Have you tried .flatten()
that's the right answer!
Paolo Di Tommaso
@pditommaso
Mar 20 2018 13:32
@rohanshah regarding your question, unfortunately that's a pattern not supported
though it could be implemented with the onError handler, it would required a bit more custom code
Maxime Garcia
@MaxUlysse
Mar 20 2018 14:00
@pditommaso I encountered a strange docker problem when using a container that was build with a change of user inside
but I'm thinking right now that it might be related to the face that I'm using the fixOwnership option
Paolo Di Tommaso
@pditommaso
Mar 20 2018 14:01
try to add `docker.runOptions="-u \$(id -u):\$(id -g)" in the config
Maxime Garcia
@MaxUlysse
Mar 20 2018 14:04
Good, I don't have to change the user in the container anymore
I should have though of that
Thanks
Luca Cozzuto
@lucacozzuto
Mar 20 2018 14:58
thank you!
PS: I found flatten a bit counter-intuitive in this case :)
Paolo Di Tommaso
@pditommaso
Mar 20 2018 15:02
life is counter-intuitive most of the times :satisfied:
Luca Cozzuto
@lucacozzuto
Mar 20 2018 15:04
so there should be a big Paolo Di Tommaso that is programming it ;)
Paolo Di Tommaso
@pditommaso
Mar 20 2018 15:04
ahah
Matthieu Foll
@mfoll
Mar 20 2018 15:28

Hi all, just a quick tip that might be useful to other users: since version 0.28.0 nextflow is using /dev/shm to create temporary files. On some of our machines /dev/shm was not 777, causing nextflow to crash with something like:

mktemp: failed to create directory via template `/dev/shm/nxf.XXXXXXXXXX': Permission denied
mkfifo: cannot create fifo `/.command.out': Permission denied
mkfifo: cannot create fifo `/.command.err’: Permission denied

It seems to be a relatively common issue on some Linux systems, and doing a chmod 777 /dev/shm (on all compute nodes if you are using an HPC cluster) is an easy fix!

Paolo Di Tommaso
@pditommaso
Mar 20 2018 15:29
too bad !
have you any chance to check what the permissions you had ?
Matthieu Foll
@mfoll
Mar 20 2018 15:31
[follm@cn02 ~]$ ls -ld /dev/shm
drwxr-xr-x 2 root root 140 Mar 20 16:08 /dev/shm
Paolo Di Tommaso
@pditommaso
Mar 20 2018 15:32
ah, it belongs to root
weird in any case, I need to check
you may want to open an issue for that
Matthieu Foll
@mfoll
Mar 20 2018 15:35
It seems to be more a problem with our linux install no, not really a nextflow issue?
Paolo Di Tommaso
@pditommaso
Mar 20 2018 15:35
I would say so, but it would be nice to prevent it in any case
Matthieu Foll
@mfoll
Mar 20 2018 15:37
ok
Vladimir Kiselev
@wikiselev
Mar 20 2018 17:04
what is the syntaxis to convert task memory to GigaBytes, i.e. an equivalent of task.memory.toBytes()
?
Paolo Di Tommaso
@pditommaso
Mar 20 2018 17:05
.toGiga()
Vladimir Kiselev
@wikiselev
Mar 20 2018 17:05
thanks!
Paolo Di Tommaso
@pditommaso
Mar 20 2018 17:05
in the latest version also just .giga
Vladimir Kiselev
@wikiselev
Mar 20 2018 17:05
:+1: thanks a lot
Vladimir Kiselev
@wikiselev
Mar 20 2018 22:22
Is it possible to use fromFilePairs() in the output of the process?
Paolo Di Tommaso
@pditommaso
Mar 20 2018 22:29
umm, no
but why use it there ?
Vladimir Kiselev
@wikiselev
Mar 20 2018 22:33
yes, I agree, I misunderstood the problem, it’s simpler than I thought. Thanks!
Paolo Di Tommaso
@pditommaso
Mar 20 2018 22:34
:ok_hand:
:wave: