These are chat archives for dereneaton/ipyrad

3rd
May 2017
susanmiller-uct
@susanmiller-uct
May 03 2017 07:29
@isaacovercast thanks for your reply- I would also have thought I could run it on my laptop (Intel i7 with 8GB RAM), however, it completely takes over my computer and runs forever. I am worried that I have something wrong in my param file? However the only things I changed were: the assembly name in line [0], the path to my files on line [4] - they are already sorted and barcodes removed, the datatype (mine are ddrad) on line [7], the restriction overhang in line [8] (to TGAC, AATT) and the output files I want in line [27]. Step three just keeps going and going and going - I can see that temporary files are being made and it does eventually get there for some of the individuals, but I have not managed to get to the end of this step as it takes too long, hence trying to get it to work on the HPC. Is this normal? Any suggestions? Also, is there a way to restrict the number of cores that it uses (I have 4 and it seems to automatically use all of them, meaning I can't do anything else on my computer while it is running as it slows down to a crawl). I know there was a way to do this in pyrad from my limited experience in pyrad before discovering ipyrad. Thanks again for your help.
Isaac Overcast
@isaacovercast
May 03 2017 12:12
@susanmiller-uct Well, it depends on how long you mean by "forever". It also depends on what your laptop looks like (how much RAM). On a computer with few resources I would expect an assembly to take many hours. If there is very little RAM it could take a day or more. My suggestion is to either find a computer with more cores and more RAM or just to wait it out. You can restrict the number of cores ipyrad uses with the -c flag (i.e. -c 3 will use 3 cores).
susanmiller-uct
@susanmiller-uct
May 03 2017 12:42
@isaacovercast thanks Isaac. I am working with our HPC person here to try to get it running on the HPC. It's a SLES11 SP4, Torque 4.1.4, Maui 3.3.1 with OpenMPI is 1.10.1 a. Anaconda is 2.7 and ipyparallel is 6.0.2 and he has given it access to 64 cores and 128GB of RAM but it's only using about 8 cores and about 2GB of RAM. We have added -c 64 to the end of the script so it has access to all of them. It has been running for over 12 hours now and it's still on step 3. Is this normal?
Isaac Overcast
@isaacovercast
May 03 2017 13:00
@susanmiller-uct What constitutes "normal" is relative to your system and your data. How long are the reads? They are single end?
Look in the _edits and _clust directories to see if files are being updated. Sometimes assemblies take a little while.
susanmiller-uct
@susanmiller-uct
May 03 2017 13:19
raw reads range from 4-10 million per file and they are from ddrad. There are finally some files being created in the _tmpalign directory so I think we are making progress! Thanks for your help!