These are chat archives for dereneaton/ipyrad

30th
Mar 2016
Isaac Overcast
@isaacovercast
Mar 30 2016 01:30
I have access to a beefy cluster (apparently not as beefy as i thought). Running on full plate of GBS data step 3 sometimes dies with the message "Killed". This is during late stage step 3, multi_muscle_align, building the clustS.gz files. Did some digging and it turns out the system is running out of memory and then force killing the 'ipyrad' process, which does end the run, but leaves ipcluster and engines orphaned. When the system auto-kills ipyrad it sends a SIGTERM. I think I can write a handler for sigterm to allow ipyrad to shut down gracefully and clean up the ipy processes.
Deren Eaton
@dereneaton
Mar 30 2016 15:03
fyi, I'm giving a workshop tomorrow at Rancho Santa Ana Botanic Garden. So hopefully the latest conda mac release is in working order.
Isaac Overcast
@isaacovercast
Mar 30 2016 18:05
Cool man that should be super fun! I just tested the current mac build on the PE and SE sim data and they both worked flawlessly.
Isaac Overcast
@isaacovercast
Mar 30 2016 18:53
Do you see step3 using a TON of memory during multi_muscle_align? I have a 40 core box and running with -c 40 on the gbs data causes an out of memory exception, kills ipyrad bcz it's using 120GB of main memory, have you seen this behavior?
Deren Eaton
@dereneaton
Mar 30 2016 19:17
No, that's weird. Memory overhead there should be tiny.
Deren Eaton
@dereneaton
Mar 30 2016 19:26
I just submitted that big paper I've been working on. So I can finally get back to coding.
Isaac Overcast
@isaacovercast
Mar 30 2016 21:18
 1454 ioverca+  20   0 1354296 614648  17884 S   1.7  0.5  39:56.29 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   1.7  0.5  39:56.34 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   1.7  0.5  39:56.39 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   0.0  0.5  39:56.39 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   1.7  0.5  39:56.44 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   0.3  0.5  39:56.45 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   4.3  0.5  39:56.58 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   6.3  0.5  39:56.77 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   6.6  0.5  39:56.97 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   6.6  0.5  39:57.17 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   7.0  0.5  39:57.38 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   6.6  0.5  39:57.58 ipyrad
 1454 ioverca+  20   0 1354296 614648  17884 S   6.3  0.5  39:57.77 ipyrad
 1454 ioverca+  20   0 1354360 614680  17884 R  40.1  0.5  39:58.98 ipyrad
 1454 ioverca+  20   0 1354360 614680  17884 R  98.0  0.5  40:01.94 ipyrad
 1454 ioverca+  20   0 1354360 614680  17884 R  98.0  0.5  40:04.90 ipyrad
 1454 ioverca+  20   0 1296740 557092  17884 R  91.7  0.4  40:07.67 ipyrad
 1454 ioverca+  20   0 1410680 671000  17884 D  77.2  0.5  40:10.00 ipyrad
 1454 ioverca+  20   0 1429592 689864  17884 R  49.7  0.5  40:11.50 ipyrad
 1454 ioverca+  20   0 1866020 1.074g  17884 R 105.3  0.9  40:14.67 ipyrad
 1454 ioverca+  20   0 2473200 1.653g  17884 R 104.0  1.3  40:17.81 ipyrad
 1454 ioverca+  20   0 3099356 2.250g  17884 R 104.0  1.8  40:20.95 ipyrad
 1454 ioverca+  20   0 3364972 2.504g  17884 R 104.3  2.0  40:24.09 ipyrad
 1454 ioverca+  20   0 4010100 3.119g  17884 R 104.3  2.5  40:27.23 ipyrad
 1454 ioverca+  20   0 4655228 3.734g  17884 R 103.3  3.0  40:30.36 ipyrad
 1454 ioverca+  20   0 5300356 4.349g  17884 R 104.3  3.5  40:33.50 ipyrad
 1454 ioverca+  20   0 5926512 4.946g  17884 R 103.3  3.9  40:36.62 ipyrad
 1454 ioverca+  20   0 6590616 5.573g  17884 R 104.3  4.4  40:39.76 ipyrad
 1454 ioverca+  20   0 7216768 6.177g  17884 R 105.0  4.9  40:42.92 ipyrad
 1454 ioverca+  20   0 7482408 6.430g  17884 R 130.5  5.1  40:46.86 ipyrad
 1454 ioverca+  20   0 7482408 6.434g  17884 R 109.3  5.1  40:50.16 ipyrad
 1454 ioverca+  20   0 7482408 6.435g  17884 R 105.3  5.1  40:53.33 ipyrad
 1454 ioverca+  20   0 7823948 6.760g  17884 R 107.6  5.4  40:56.59 ipyrad
 1454 ioverca+  20   0 8374204 7.270g  17884 R 107.6  5.8  40:59.84 ipyrad
 1454 ioverca+  20   0 9000360 7.882g  17884 R 105.3  6.3  41:03.02 ipyrad
 1454 ioverca+  20   0 9607536 8.462g  17884 R 107.6  6.7  41:06.28 ipyrad
 1454 ioverca+  20   0 9975.3m 9.041g  17884 R 104.3  7.2  41:09.44 ipyrad
 1454 ioverca+  20   0 10.248g 9.548g  17884 R 108.3  7.6  41:12.70 ipyrad
 1454 ioverca+  20   0 10.248g 9.548g  17884 R 110.9  7.6  41:16.05 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 109.9  7.6  41:19.37 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 107.0  7.6  41:22.59 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 104.6  7.6  41:25.76 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 108.0  7.6  41:29.01 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 105.3  7.6  41:32.19 ipyrad
 1454 ioverca+  20   0 10.248g 9.549g  17884 R 107.9  7.6  41:35.45 ipyrad
Sorry for the gitterspam, this is output of top | grep ipyrad for step3, once cpu starts to spike the ram goes bonkers. I'm assuming this is inside sample_cleanup(), since this is the only section of code that's run by the ipyrad thread at this point... i'm investigating still..