Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 06 14:41
    ewels opened #1405
  • Dec 06 10:09
    pditommaso commented #946
  • Dec 06 07:03
    kemin711 opened #1404
  • Dec 05 11:58
    ii-bioinfo-thomas opened #1403
  • Dec 05 10:22
    ii-bioinfo-thomas edited #1402
  • Dec 05 10:20
    ii-bioinfo-thomas opened #1402
  • Dec 05 09:10
    ulfschaefer opened #1401
  • Dec 05 07:50
    pditommaso commented #1400
  • Dec 04 23:14
    durrantmm edited #1400
  • Dec 04 23:13
    durrantmm opened #1400
  • Dec 04 18:32
    pditommaso labeled #1399
  • Dec 04 16:55
    ii-bioinfo-thomas edited #1399
  • Dec 04 16:54
    ii-bioinfo-thomas edited #1399
  • Dec 04 16:52
    ii-bioinfo-thomas opened #1399
  • Dec 04 15:38
    sb10 commented #1088
  • Dec 04 15:07
    pditommaso commented #1088
  • Dec 04 15:04
    pditommaso edited #1398
  • Dec 04 13:12
    sb10 commented #1088
  • Dec 04 12:38
    jma1991 commented #1387
  • Dec 04 11:20
    sb10 commented #1088
Paolo Di Tommaso
@pditommaso
can't check now, you may wont to report an issue
Daniel E Cook
@danielecook
I can do that
thanks
Paolo Di Tommaso
@pditommaso
welcome
Maxime Garcia
@MaxUlysse
OK, so I made a mistake earlier, it is not working in fact
I'll make a minimal example and an issue
Sri Harsha Meghadri
@harshameghadri
Hey folks, I am pretty new to using docker and singularity. I want to use the nf-core/rnaseq for my analysis. I consistently get errors while trying to execute this command singularity pull --name nf-core-rnaseq-1.3.img docker://nf-core/rnaseq:1.3
Unable to pull docker://nf-core/rnaseq:1.3: conveyor failed to get: Error reading manifest 1.3 in docker.io/nf-core/rnaseq: errors:
I am trying to pull to rackham. My analysis needs to be executed on bianca. Any tips are super appreciated.
Maxime Garcia
@MaxUlysse
try singularity pull --name nf-core-rnaseq-1.3.img docker://nfcore/rnaseq:1.3 instead, without the - in nf-core
I'm guessing if you have more question about nf-core pipelines, it should be better on our slack: https://nf-co.re/join
Sri Harsha Meghadri
@harshameghadri
I tried that as well getting the same error @MaxUlysse
Maxime Garcia
@MaxUlysse
You tried that on bianca?
Sri Harsha Meghadri
@harshameghadri
nope on rackham, I guess it needs internet
Maxime Garcia
@MaxUlysse
Sure
Just trying to find an easy mistake, sorry ;-)
Have you tried running that on an interactive node?
Sri Harsha Meghadri
@harshameghadri
hmmm on rackham? I dnt have allocation there.
Maxime Garcia
@MaxUlysse
I'm afraid singularity might be too demanding on the regular login node
Let me see if I can help you in an other way
Sri Harsha Meghadri
@harshameghadri
perfect, thank you Maxime.
Maxime Garcia
@MaxUlysse
@harshameghadri I messaged you ;-)
Marko Melnick
@Senorelegans
Is there any way to force a process to wait for another process to finish before it is started? I tryed to do it with a dummy channel but I am concatenating files to the channel amounts change from one process to the next.
Maxime Garcia
@MaxUlysse
can't you use the output from the process that need to be finished as an input for the other?
maybe with a .collect() to be sure to catch multiple executions of said process
Marko Melnick
@Senorelegans
I am concatenating fastq files by groups (I made a parser in groovy to separate by group). Is there no way to just make one process wait for another one with a dummy channel or some null variable?
I guess my real issue is I am reading channels from pairs earlier. And I have a list of the file name with condition group in a sample table that I can read, but I am struggling bringing them and operating on them by group.
Stijn van Dongen
@micans
@Senorelegans can you make a functioning toy example that illustrates your issue? Grouping is usually done by groupKey -> transpose -> groupTuple; e.g.
Channel.from(['a', [1, 2, 3]], ['b', [4, 5]], ['c', [6, 7, 8]])
   .map { tag, stuff -> tuple( groupKey(tag, stuff.size()), stuff ) }
   .view()
   .transpose()
   .map { tag, num -> [tag, num*num+1 ] }
   .view()
   .groupTuple()
   .view()
Ashley S Doane
@DoaneAS
@rsuchecki any ideas on this issue with java cpu use when nextflow is reading the results cache following nextflow -resume? I’m running nextflow on plenty of available resources (192 CPUs, 10T ram), but I’ll try submitting nextflow command as a job. This way SGE will kill it if CPU usage is too high.
Ashley S Doane
@DoaneAS

Also, I had the

executor {
    queueSize=1000
}

But thinking more about how sge works, this is not necessary (jobs are not run based on how long they have been in queue, but based on job priority that SGE determines and updates). Seems possible that this setting coild have caused an issue.

Anthony Ferrari
@af8
Hi all, is it possible to have publishDir directive to resolve symlinks in move mode ? I have a folder going from a process to another, I modify its content in the second process and publish it. But what is actually published is the symlink to the first process workdir. I would like it to be moved physically. Thanks
Rad Suchecki
@rsuchecki
A few ideas @DoaneAS - but I am not convinced by any of them... here is a couple
  • If the problem persists try to look into JVM settings regarding memory and GC - which could be the cause - but why would there be that much garbage to collect in the first place?
  • IO - NF process having to shift all the result files to publishDir - again there is not that many of them so not sure this could be the issue but you could disable publishDir in the first place to see if it makes a difference and re-enable in another run with -resume to publish the cached results.
Ashley S Doane
@DoaneAS
@rsuchecki thanks for the suggestions!
marchoeppner
@marchoeppner
Hi, quick question about joining multiple channels (2 parts to this question):
if I have multiple output channels, each of the format [ a_label, a_result ] - would I put them together by successive ".join" statements?
input_to_summary = read_files_summary .join(fusioncatcher_fusions) .join(star_fusion_fusions) .join(ericscript_fusions) .join(pizzly_fusions) .join(squid_fusions)
or is there a more elegant way?
Second part, what would be a way to deal with empty channels in this scenario?
scenario is the following: multiple parallel analyses of the same sequencing data, which are then merged into a report - each analysis being optional.
Stijn van Dongen
@micans
@marchoeppner I don't see anything more elegant; to solve the empty channels I can only think of using mix() combined with groupTuple(), e.g.
a = Channel.from(['a', 1], ['b', 2])
b = Channel.from(['a', 3], ['b', 4])
c = Channel.empty()

a.mix(b).mix(c).groupTuple().view()
marchoeppner
@marchoeppner
ok thanks, I will try that!
Stijn van Dongen
@micans
@marchoeppner note that groupTuple() will block until the channel has completed. In your case, you may know the size of each eventual tuple (as it is the number of analyses), so you could give it the size parameter, in that case a tuple is released once it has that size.
Stijn van Dongen
@micans
@marchoeppner further caveat; the tuples you get can have the analyses in different orders. If the elements are files I imagine it does not matter much.
marchoeppner
@marchoeppner
that might actually be a problem, since I need to do something like:
input: set val(sample_id),file(analysis1),file(analysis2),file(analysis3) from Foo
so it still is a bit tricky I reckon..
it's not my pipeline, so I don't have too much control over the basic logic of it all, just trying to implement support for multiple input data sets (right now it assumes that there is only one sample)
Stijn van Dongen
@micans

How is this

input: set val(sample_id),file(analysis1),file(analysis2),file(analysis3) from Foo

going to work if some analysis could be missing? As for the order that could be fixed I think with an additional sorting step.

marchoeppner
@marchoeppner
indeed ^^
it used to be multiple arguments under "input:" with an added ".ifEmpty('')" - but that seems difficult to do now