These are chat archives for dereneaton/ipyrad

30th
Aug 2018
Amanda Haponski
@ahaponski_twitter
Aug 30 2018 17:54
Hi @isaacovercast !!! I have a question about speeding up Step 3. I've got paired ddRAD data for ~10 individuals with 5-10 million reads per individual. Right now I'm at 9 days and have only reached 94% clustering. I am running on a single node with 16 clusters at 3800mb (University system limit). Is there anyway to speed this up besides adding more nodes? The reason I ask is that I have another project with ~30 individuals and ~10-15 million reads per individual that I need to process. Thanks in advance.
Isaac Overcast
@isaacovercast
Aug 30 2018 21:10
@Cn_lvnt_twitter Did this happen during step 7 or during step 6? Did you create a branch and subsample the data? Can you rerun the step that crashed and include the -d flag and look at the last 20 or so lines of the ipyrad_log.txt and see if there's anything unusual?
@danielyao12 Those messages aren't very informative, it seems. Can you rerun step3 with the -d flag and email me the ipyrad_log.txt file?
@danielyao12 This is probalby a problem with the dataset, but its's somewhat hard to say from the error messages here.
Isaac Overcast
@isaacovercast
Aug 30 2018 23:07
@ahaponski_twitter Hi Amanda. Is this correct that your university HPC limits RAM to 3800mb (i.e. ~4Gb)? This is probably what's causing the holdup, we recommend 4GB per core, and you've got 4GB for 16 cores. If you can't get more ram you could try reducing the number of cores, this could speed it up (if it's running out of RAM, then reducing the number of cores will reduce ram usage and reduce paging, which would make it go faster).
If it's just 10 samples you could probably just as easily do this assembly on your laptop.