These are chat archives for dereneaton/ipyrad

27th
Oct 2016
Emily Warschefsky
@ewarschefsky_twitter
Oct 27 2016 14:58
@dereneaton @isaacovercast I am having the same issue as @slipshut. I only use conda for ipyrad and just ran the updates for both conda and ipyrad before starting the analysis. Is it possible it is a memory issue? When I look at the log file from debug mode it says:
2016-10-27 05:06:26,625     pid=4997     [cluster_across.py]    INFO     single cat here
2016-10-27 05:06:27,116     pid=4950     [cluster_across.py]    ERROR      error in singlecat (YA) MemoryError()
2016-10-27 05:06:27,132     pid=4950     [assembly.py]    INFO     tuple index out of range
2016-10-27 05:06:27,136     pid=4997     [cluster_across.py]    INFO     single cat here
2016-10-27 05:06:27,189     pid=4950     [assembly.py]    INFO       shutting down engines
2016-10-27 05:06:27,272     pid=4997     [cluster_across.py]    INFO     single cat here
I previously used the same datafile and had no issue when I ran it on 32 cores instead of 16, with more memory.
Deren Eaton
@dereneaton
Oct 27 2016 15:51
@ewarschefsky_twitter @slipshut hmm, I guess the memory requirements increases quite a bit with longer reads, I've done most memory testing with single-end 100 bp reads. I'm hoping to optimize this eventually to avoid memory limits all together, but for now I can make a quick fix that will limit the number of simultaneous jobs during the most mem intensive part of step 6 so that it does not hit a limit.
James Clugston
@Cycadales_twitter
Oct 27 2016 17:59
image001.png
@dereneaton @isaacovercast see above. any ideas on this error? I am having no luck in getting any samples past this step now in the past month. I am really not sure what to do here.
Deren Eaton
@dereneaton
Oct 27 2016 18:52
@Cycadales_twitter Can you send me your .json file and four of your clustS.gz files to deren.eaton@yale.edu. I'll try to figure it out.
James Clugston
@Cycadales_twitter
Oct 27 2016 19:10
@dereneaton yea I can do that tomorrow no worries. But I do not have access to the server from here.
Emily Warschefsky
@ewarschefsky_twitter
Oct 27 2016 20:57
@dereneaton - okay that would make sense - I am running PE 150.
Shea Lambert
@SheaML
Oct 27 2016 23:45
@dereneaton @isaacovercast I'm seeming to have some trouble getting ipyrad to parallelize properly on my university's HPC. I'm attempting to run step 3 using a reference assembly and ~150 samples of PE100. I'm not seeing much CPU or memory usage, at least during step 3. Other steps run much faster and I assume they might be parallelizing OK. I've tried starting ipcluster manually (with --n=28, my #cpu) using a qsub -I session and also a submit script with "ipyrad ... -c 28 –MPI." Thanks for any suggestions.