These are chat archives for dereneaton/ipyrad

5th
Feb 2016
Isaac Overcast
@isaacovercast
Feb 05 2016 02:42
numba bug is fixed dereneaton/ipyrad#100
What's your comfort level on just enforcing conda based install. pip is fucked for mpi4py AND numba. Fixing either of those for pip install would be a serious bitch.
I don't think the miniconda install is too egregious.
Deren Eaton
@dereneaton
Feb 05 2016 03:10
I agree. Let's plan to very strongly encourage conda. I think we can write around numba so that it's not used if not present, and we could write pip instructions as backup which also explain that you need to install mpi, hdf5, etc. 🧀
Deren Eaton
@dereneaton
Feb 05 2016 07:02
the 10 second wait before timing out in the _launch command doesn't appear to be long enough on HPC clusters, or at least not on the old dog I'm using right now. It took 28 seconds to spin up the connection :confounded:
Deren Eaton
@dereneaton
Feb 05 2016 17:49
do you have time to chat later today? I want to discuss prioritization a bit. Maybe a quick google hangout.
Deren Eaton
@dereneaton
Feb 05 2016 18:11
The things I am planning to focus on immediately are: (1) make sure pair-merging and concatenation is correct, (2) make HDF5 arrays chunkable for steps 6-7, (3) finish loci2* conversions and vcf output.
Also: (4) Docs, (5) HPC testing.
Deren Eaton
@dereneaton
Feb 05 2016 21:22

I'm thinking again about whether different assembly method names would be more clear. denovo and denovo_only both sound like they are just doing denovo.

Maybe the following four names would be more clear. The latter three all make it clear that they require a reference sequence.
1) denovo
2) reference (this is the former reference_only)
3) reference_add (this is the former hybrid, it adds denovo to ref)
4) reference_sub (this is the former denovo_only, it subtracts denovo from ref)

What do you think?

Isaac Overcast
@isaacovercast
Feb 05 2016 22:05
Sorry for the delayed response, i'm on my way to florida for a couple days. I'm on the airplane now, so have about an hour if you want to check in
I agree that the names could be more clear. I had to explain them to glenn and i realize there's a ton of ambiguity.
Deren Eaton
@dereneaton
Feb 05 2016 22:06
Cool. join a chat here: appear.in/massive-zebra
Isaac Overcast
@isaacovercast
Feb 05 2016 22:09
oh crap, can we just chat here, i'm on a plane and don't want to be talking to my computer like a newb.
i can video chat later, after i get off the plane.
Deren Eaton
@dereneaton
Feb 05 2016 22:09
oh, yeah, sure
Isaac Overcast
@isaacovercast
Feb 05 2016 22:09
The conda people patched the solver and pushed a test build.
Deren Eaton
@dereneaton
Feb 05 2016 22:10
just saw that, I hope it helps the speed
Isaac Overcast
@isaacovercast
Feb 05 2016 22:10
THey used ipyrads build problem for profiling: "@isaacovercast, I used your exact issue in my profiling iterations "
Deren Eaton
@dereneaton
Feb 05 2016 22:10
sweeeeet
Isaac Overcast
@isaacovercast
Feb 05 2016 22:10
Lol, yeah it works hella better, it's instantaneous
Deren Eaton
@dereneaton
Feb 05 2016 22:10
damn, we should start writing to everyone with our problems
Isaac Overcast
@isaacovercast
Feb 05 2016 22:11
Ok, yeah priorities
lol i know!
Deren Eaton
@dereneaton
Feb 05 2016 22:11
the new ipyparallel is on pip, but not yet on conda. I've been testing it.
super easy to connect to multiple nodes using MPI now, no need for manually editing files and shit.
yeah, what are you thinking for priorities?
Isaac Overcast
@isaacovercast
Feb 05 2016 22:18
Well getting the conda install sorted out has been a big priority, so that kinda clears my plate a little. 1) I'm trying to track down a nasty bug in glenn's gbs data that crashes step3. 2) Tuning reference sequence mapping parameters to get good performance. Right now I'm using the clustering parameter for for the reference mapping sequence identity parameter, and i'm not sure that's the best choice. 3) Definitely docs, i have a couple reference seq mapping workflows that i want to have good docs for.
Deren Eaton
@dereneaton
Feb 05 2016 22:19
Sounds great. And yeah, great job on the conda stuff.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:19
I will be doing HPC testing on my end as well. Also, i've only been doing refseq testing on gbs, so i want to do a full test run on the ddrad you sent along, that's definitely on my plate.
Deren Eaton
@dereneaton
Feb 05 2016 22:20
are there instructions somewhere for how I can upload to anaconda?
or does it take the version straight from github?
Isaac Overcast
@isaacovercast
Feb 05 2016 22:20
Hm, i think i documented it, let me look.
I didn't doc it, i'll write it up in the ticket.
echo "anaconda_upload: yes" >> ~/.condarc
Deren Eaton
@dereneaton
Feb 05 2016 22:24
cool.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:24
cd ipyrad; conda build conda.recipe
uploads automatically. but i still have to figure out how to set it to default to the new version. as it is i still have to go to anaconda.org and disable/delete the old version. Kind of annoying.
What do you think about this galaxy thing?
Deren Eaton
@dereneaton
Feb 05 2016 22:26
I don't know that much about it...
but it seems to be Python related, so I think they should have no problem installing ipyrad
apparently stacks is on it, and gets used pretty often that way. So I'm all for it.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:27
oh ps - re: conda, before you do conda build you have to do anaconda login
username is ipyrad, password is aardvark1
Deren Eaton
@dereneaton
Feb 05 2016 22:27
gotit
actually I'm getting an invalid user/pass error
Isaac Overcast
@isaacovercast
Feb 05 2016 22:29
lol password actually is watwatwat
I know we talked about mid february being the "deadline" for getting it settled. Should we firm that up a little bit? say feb 19th? two weeks from today? Think that's doable?
Deren Eaton
@dereneaton
Feb 05 2016 22:35
Yeah, that would be awesome.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:36
Ok cool. I think a big two week push should get us pretty fuckin close.... you think it's worth migrating docs to readthedocs at this point or should we leave them in github wiki.
Deren Eaton
@dereneaton
Feb 05 2016 22:37
Let's try to get everything relevant into rtd so it's at least there, and we can polish it up as we go.
I'll hit it hard on steps 6-7 the next few days. I'm still not sure the paired reads are lined up right, I need to get back to testing that on real data too.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:39
Sounds good. oof, i have never used rtd before, it's not exactly intuitive, or my intuition needs adjustment. You make docs in github and then import them into rtd? is that right?
Deren Eaton
@dereneaton
Feb 05 2016 22:40
I was thinking it would be really great if we could tag the reference reads with some kind of locus name that indicated which reference it hit to, and where. That info is in the bam files, but we don't carry it through to the end currently. Not sure how much work that is, maybe a v.2 effort.
Isaac Overcast
@isaacovercast
Feb 05 2016 22:40
I totally agree, that's something i was thinking as well.
Deren Eaton
@dereneaton
Feb 05 2016 22:40
you just have to put .rst files in the docs/ dir
which is pretty similar to markdown format
the docs/index.rst creates a page hierarchy
Isaac Overcast
@isaacovercast
Feb 05 2016 22:41
Mostly i've been hesitant about it because the name line gets munged in some many different places i haven't figured out a good way to work it in that would not either a) get sliced off at some point or b) disrupt downstream analysis. I just need to sit down and thinka bout it for five minutes.
Deren Eaton
@dereneaton
Feb 05 2016 22:42
I think the docs auto-build after each push to github
Isaac Overcast
@isaacovercast
Feb 05 2016 22:42
Wow, the future IS now.
:rocket:
Deren Eaton
@dereneaton
Feb 05 2016 22:42
lol
Isaac Overcast
@isaacovercast
Feb 05 2016 22:43
alright man, my battery is about to keel over, i better close up shop for now. I'll check in with you later, but it sounds like we got a good plan, sounds doable.
Deren Eaton
@dereneaton
Feb 05 2016 22:43
cool. Have a good trip :palm_tree: :sun:
Isaac Overcast
@isaacovercast
Feb 05 2016 22:44
thx. later.
Deren Eaton
@dereneaton
Feb 05 2016 22:44
1/2 on emojis