These are chat archives for nextflow-io/nextflow

20th
Sep 2018
Karin Lagesen
@karinlag
Sep 20 2018 07:12
I have had problems with processes having data sets that are linked to the wrong datasets from previous steps
I finally realized that this is only an issue for the proceesses that I have that receive input from two different preceeding processes
so I am then assuming that this error occurrs because the channels are FIFO queues and that I am just receiving data sets in random order across the channels
an example is then this input:
set pair_id, file("${pair_id}_mapped_sorted.bam"), file("${pair_id}_mapped_sorted.bam.bai") from bwamem_results
set pair_id, file("${pair_id}_spades_scaffolds.fasta") from assembly_results
thus, how can I specify that the pair_ids in this case should be the same?
Luca Cozzuto
@lucacozzuto
Sep 20 2018 07:30
you should merge them
Karin Lagesen
@karinlag
Sep 20 2018 07:30
...what would that look like then...?
Luca Cozzuto
@lucacozzuto
Sep 20 2018 07:30
one sec
:)
Karin Lagesen
@karinlag
Sep 20 2018 07:33
awesome, thanks!
have to admit that it took me too many hours to remember that these were fifo queues....
Luca Cozzuto
@lucacozzuto
Sep 20 2018 07:34
set pair_id, file(sortedbam), file(baifile) , file(scaffolds) from bwamem_results.join(assembly_results)
@karinlag I think we all had the same problem at certain point :)
I just changed because you don't need that sintax you can use the variables directly
Karin Lagesen
@karinlag
Sep 20 2018 07:51
@lucacozzuto even better, thanks so much!
misssoft
@misssoft
Sep 20 2018 11:16
Hello, wonder if there are any examples/tutorials running Nextflow to a Kubenetes cluster? Or anyone can help on questions:
Questions related to K8s, 1. Is nextflow kube run only support the nextflow hosted in Github? Can it support Singularity? How to prepare the singularity image in the shared Persistent volumn? How to get the input files to the kube cluster? Thanks!
Has anyone run Nextflow pipelines on Azure Kubernetes Service (AKS)?
Alexander Peltzer
@apeltzer
Sep 20 2018 12:04
Does the groupBy method work with a regular expression? Want to group samples based on their lane identifiers....
e.g. Channel.groupBy('^.*_(L[0-9]+)_.*\.fastq\.gz') ?
Paolo Di Tommaso
@pditommaso
Sep 20 2018 12:13
do you mean groupTuple ?
Alexander Peltzer
@apeltzer
Sep 20 2018 12:15
Well, I'd like it to work with Single End and Paired End data ideally... so groupTouple might be better?
Paolo Di Tommaso
@pditommaso
Sep 20 2018 12:15
I guess so, it should also support regexp
something like .groupTuple(~/^.*_(L[0-9]+)_.*\.fastq\.gz/)
Alexander Peltzer
@apeltzer
Sep 20 2018 12:17
I'll test this - would be perfect if that works that nicely to be able to merge together samples from multiple lanes efficiently, no matter if they are SE or PE sequenced
Thanks, I'll report back :-
:-)
Paolo Di Tommaso
@pditommaso
Sep 20 2018 12:18
@misssoft there's @mes5k has some experience with that, there should be a thread in this channel some days/weeks ago
misssoft
@misssoft
Sep 20 2018 14:36
Thank you very much @pditommaso.
Paolo Di Tommaso
@pditommaso
Sep 20 2018 14:37
hope it helps
misssoft
@misssoft
Sep 20 2018 15:07
@pditommaso tried to find related thread. I guess there is no tutorial/example running a pipeline in a Kubenetes cluster?
Paolo Di Tommaso
@pditommaso
Sep 20 2018 15:11
I was referring to this :point_up: August 17, 2018 8:17 PM
the point is the configuration of EFS storage as pvc
misssoft
@misssoft
Sep 20 2018 15:17
@pditommaso thank you, I am trying it on Azure Kubenetes Service (AKS), the pvc is available to use in a Pod, but not sure how Nextflow get it configured... will read the thread.... thank you again!
Paolo Di Tommaso
@pditommaso
Sep 20 2018 15:18
then you have done
misssoft
@misssoft
Sep 20 2018 16:14
@mes5k hello Mike, can I ask you a quick question in terms of Nextflow on Kube? Would you mind to share the command you run "nexflow kuberun", I have trouble to locate the pipeline in Github, and the parameter of -v "-v my-pvc:/mnt/path", do you need to set up the /mnt/path? thank you.
Mike Smoot
@mes5k
Sep 20 2018 16:20

Hi @misssoft, here's the command I ran:

nextflow kuberun http://git.l.synthgeno.global/msmoot/nextflow_test_k8s.git -v "nextflow-efs:/mnt/efs"

That's using an internal git repo. If you're having trouble with github, maybe provide the full url? And yes, I had to configure /mnt/efs. Also, I did this in AWS and not AKS, which I'm sure changes things.

misssoft
@misssoft
Sep 20 2018 16:25
thanks a lot, @mes5k, that is very helpful.