re: hackathon Saturday yeah we could do that, i don't have any specific plans. sounds like a good idea.
Where were you thinking location-wise?
Re: --mpi flag, i haven't really fucked with it that much, so it's hard to say. What little i've tried on the cuny cluster and on the lsu cluster hasn't seemed to work (for CLI), at least across cluster nodes, which is when it's really useful.
Also, question about memory usage, denovo step 3 is really 2 steps, clustering and alignment. Is your impression that clustering is much more memory intensive than alignment? That's my impression, and it raised some red flags (or maybe yellow flags) on the LSU cluster...
Cool. No idea, where's good for hacking? I haven't tried large connections with MPI recently, it's probably not up to par yet.
alignment is low memory, the speed is limited by our inefficient I/O to muscle. Vsearch is pretty low memory too, unless you give it unlimited threads. But maybe if too many are running concurrently the memory gets too high. I suppose in the case that is the problem the user should just use fewer cores. I've been thinking too that it would be kind of nice to just make the aligning a separate step instead of this state=2.5 business.
Loading the utemp into memory to build the clusters could crank up memory too, but I would expect that problem to occur in step6 more dramatically than step3.
re: saturday, idk, somewhere with lots of megahertz. And coffee. There's a couple coffee shops downtown that are decent. Could also work at the lab at city college, or at the grad center (midtown). lots of options..