These are chat archives for dereneaton/ipyrad

14th
Jun 2018
Todd Pierson
@twpierson
Jun 14 2018 13:57
@isaacovercast and @dereneaton, a quick question: is the Dfoil test implemented in ipyrad yet?
Saritonia
@Saritonia
Jun 14 2018 14:10
Dear @isaacovercast and @dereneaton. Hope you are well. I have gotten to finish the step 7 with a large dataset and the results were seemingly successful. However, some formats were no created: .u.snps.phy, .geno, .u.geno, .gphocs, .str, .u.str.str and .nex. I have read in a previous message that it could be solved by updating the dependencies. However, I have used last version of ipyrad (v.0.7.24.) in that step. There is a binary file with the format .hdf5 in the outfile folder (also in the _across folder) which it should not be generated. What is this file? What could be the reason because all the files are not created?. On the other hand, I have seen that there is no a file with the information of the partition of the loci as .phy.partition in Pyrad. How can I create this file? Thanks you very much in advance for your help!!!
J. Mark Porter
@jmark_porter_twitter
Jun 14 2018 15:28

Hi @isaacovercast, I don't think that ipyrad has completely stalled, but is instead running very slowly. There are still two recalcitrant samples of the 12 test samples that have not chunked sequences or finished the clustering of step 3. At the same time there is no evidence of any error that I can see. Here is the progression of one of the two recalcitrant samples during clustering:

on June 12:
-rw-r--r-- 1 mporter columbuslab 0 Jun 12 18:37 15421_I.htemp
-rw-r--r-- 1 mporter columbuslab 0 Jun 12 18:37 15421_I.utemp

on June 13:
-rw-r--r-- 1 mporter columbuslab 16777216 Jun 13 00:43 15421_I.htemp
-rw-r--r-- 1 mporter columbuslab 33554432 Jun 13 03:36 15421_I.utemp

on June 14
-rw-r--r-- 1 mporter columbuslab 16777216 Jun 13 00:43 15421_I.htemp
-rw-r--r-- 1 mporter columbuslab 67108864 Jun 14 02:36 15421_I.utemp

It looks like the algorithm is running, albeit slowly, and writing to file at least once every 24 hours. I was concerned that there might be a problem with openmpi (the tasks not being partitioned properly), so I checked the job status (scontrol show job) and everything appears to be setup correctly and running with 800GB. The data set assembles without problems using the STACKS pipeline, so I do not think that it is something about the original data files. Should I update ipyrad to the most current version and try again? Thanks for your help.

Isaac Overcast
@isaacovercast
Jun 14 2018 15:35
@jmark_porter_twitter It can't hurt to upgrade. This is just really unusual. It's very conspicuous that the htemp file doesn't change, as this is recording matching seeds. Can you email me the ipyrad_log.txt file? iovercast@gc.cuny.edu. The other thing I was thinking is, have you tried running without mpi? If you're running 12 samples on 40 cores across 4 nodes this is introducing a lot of possible complexity. My belief is that ipyrad should handle this situation fine, but it could be that a sample is getting split across nodes in a way that's causing it to croak. It could be worth trying turning off mpi and just assembling on 1 node with 10 cores.
@jmark_porter_twitter When you say 'running with 800GB' do you mean RAM?
@Saritonia What are you using for your output_formats parameter? If you use * it will create all output formats we support. If it's not creating these then check the ipyra_log.txt file for any error messages. The hdf5 file in the _across directory is an intermediate file that we use for storing clustered locus data, it's necessary for ipyrad to run step 7.
@jmark_porter_twitter I created an issue for this to keep track of details: #300
J. Mark Porter
@jmark_porter_twitter
Jun 14 2018 15:40
Hi @isaacovercast, yes, I was referring to the total RAM allocated to the job. I will send you the entire ipyrad_log.txt file, then kill the run. Before I upgrade I will try running on a single node with 10 nodes, each with 20GB.
Isaac Overcast
@isaacovercast
Jun 14 2018 16:28
@jmark_porter_twitter Yeah it might save us some time if you want to dropbox me the fastq files for the 12 samples you're using. If I can get a look at the data maybe I can figure out if it's something with the data or if it's the mpi/cluster config.
Saritonia
@Saritonia
Jun 14 2018 17:11
@isaacovercast Thanks you Isaac. I use to create all output formats. I have already check the ipyrad_log.txt file and any error messages were created according step 6 and 7. In the _across folder there are these files: clust.hdf5, utemp.sort, cat.clust.gz and s6_cluster_stats.txt. I could also run step 7 using this _across folder as input and the few formats created are ok (.loci, .phy, .snps.map, .snps.phy, .vcf and .stats.txt). I have even run a phylogenetic analysis and the results have sense!!. Do you have any suggestion about what is happening and how can I solve it? If you need any file to figure out the problem, let me know. Thanks for your help.
Isaac Overcast
@isaacovercast
Jun 14 2018 17:22
Can you paste your params file here?