These are chat archives for dereneaton/ipyrad

1st
Apr 2016
Isaac Overcast
@isaacovercast
Apr 01 2016 17:35
Ooof, i'll work on the refmap error, wat the fuck... I'll try to fix it today, sorry it bombed in the tutorial, i swear i tested it. Which dataset was this?
Isaac Overcast
@isaacovercast
Apr 01 2016 17:44
Yes, on real data i do see that some samples finish and move to state 3. I've been testing with the memory_profiler python module and so far haven't seen anything blindly stupid. There's a part_ of me that _suspects this is related to ipyparallel 5.0...
10946 ioverca+  20   0 51.315g 0.050t   2664 S  16.0 40.4  17:31.66 python                                                                           
10946 ioverca+  20   0 46.177g 0.045t   2664 S  16.7 36.3  19:12.17 python                                                                           
10946 ioverca+  20   0 43.249g 0.042t   2664 S  16.9 34.0  20:53.40 python                                                                           
10946 ioverca+  20   0 52.018g 0.050t   2664 S  18.7 41.0  22:45.89 python                                                                           
10946 ioverca+  20   0 57.738g 0.056t   2664 S  17.5 45.5  24:31.09 python
This is the controller process, you can see difference in cpu (at ~ 2 min intervals) look at that mem usage! 50GB main mem AND virtual.
Isaac Overcast
@isaacovercast
Apr 01 2016 18:31

At least for me this is because ipyrad is getting SIGKILL'd, which you can't actually catch.

And it's not saving their state in 2.5 when it crashes, ugh.

Isaac Overcast
@isaacovercast
Apr 01 2016 19:09
It's almost certainly ipyparallel. I rolled back to 4.0.1 and it fixed the problem.
Deren Eaton
@dereneaton
Apr 01 2016 20:26
OK, we should try to make a simple example snippet of this to raise an issue with them if we can. Cuz we need 5.0 for the MPI mode to work.
Isaac Overcast
@isaacovercast
Apr 01 2016 20:27
I'll work on it.
The call to map_async(muscle_align, submitted_args) is allocating a TON of mem.
Deren Eaton
@dereneaton
Apr 01 2016 20:28
The new apply (instead of map) arguments I've been working on for step3 are super sharp. I haven't had a chance yet to see if it runs into this same problem. Got it on a separate branch. But it might work better.
Isaac Overcast
@isaacovercast
Apr 01 2016 20:28
What dataset were you testing when you saw the refmap bug? I can't replicate w/ simrad_test_R1
Deren Eaton
@dereneaton
Apr 01 2016 20:29
it was with simrad_test_R1.fastq.gz using a fresh conda install
copy/pasting straight from the advanced_tutorial
Isaac Overcast
@isaacovercast
Apr 01 2016 20:29
Ok, i'll try from scratch on a clean box.
Deren Eaton
@dereneaton
Apr 01 2016 20:30
cool. we're getting there. Gotta run.
Isaac Overcast
@isaacovercast
Apr 01 2016 20:31
talk to you soon.