These are chat archives for dereneaton/ipyrad

2nd
Jan 2018
Saritonia
@Saritonia
Jan 02 2018 09:25

Hi Deren and Isaac. My name is Sara and I am a phD student from the University of Seville (Spain). First of all, congratulation for this amazing software. I started using PyRAD and I am trying apply ipyrad for a population genetics study. I am using a HPC cluster and SLURM system. I have achieved to run steps 1 and 2 trouble-free but I am having troubles to run step 3 using branching assembly with two coverage values (two parallel analyses running at the same time). A lot of errors have been written in the .out file:

Begin run: 2017-12-29 15:13
Using args {'preview': False, 'force': False, 'threads': 2, 'results': False, 'quiet': False, 'merge': None, 'ipcluster': None, 'cores': 32, 'params': 'params-data1.txt', 'branch': None, 'steps': '345', 'debug': False, 'new': None, 'MPI': False}
Platform info: ('Linux', 'trueno323.csic.es', '3.10.0-514.el7.x86_64', '#1 SMP Wed Oct 19 11:24:13 EDT 2016', 'x86_64')2017-12-29 15:15:32,871 pid=343261 [util.py] ERROR Error: ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89Mmerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpW1_Yk2_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpx41FBr_nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'] vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores
https://github.com/torognes/vsearch

Fatal error: File type not recognized.

2017-12-29 15:15:32,930 pid=343261 [util.py] ERROR Exception in merge_pairs - ('Error merge pairs:\n %s\n%s', ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89Mmerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpW1_Yk2_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpx41FBr_nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'], 'vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores\nhttps://github.com/torognes/vsearch\n\n\n\nFatal error: File type not recognized.\n')
2017-12-29 15:15:33,886 pid=343286 [util.py] ERROR Error: ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89M.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_ma_89Mmerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmp_kFWQb_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpTmpSNX_nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'] vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores
https://github.com/torognes/vsearch

Unable to open file for reading (/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_49B.trimmedR1.fastq.tmp1)

2017-12-29 18:59:41,164 pid=343215 [util.py] ERROR Exception in merge_pairs - ('Error merge pairs:\n %s\n%s', ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_49B.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_49B.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_49Bmerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmp0DBQRN_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpcgNL7T_nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'], 'vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores\nhttps://github.com/torognes/vsearch\n\n\n\nUnable to open file for reading (/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_49B.trimmedR1.fastq.tmp1)\n')
2017-12-29 19:00:00,219 pid=343175 [util.py] ERROR Error: ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266Omerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpqklxLe_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpSIBeG__nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'] vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores
https://github.com/torognes/vsearch

Unable to open file for reading (/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR1.fastq.tmp1)

2017-12-29 19:00:00,235 pid=343175 [util.py] ERROR Exception in merge_pairs - ('Error merge pairs:\n %s\n%s', ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266Omerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpqklxLe_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpSIBeG__nonmergedR2.fastq', '--fasta_width', '0', '--fastq_minmergelen', '35', '--fastq_maxns', '6', '--fastq_minovlen', '20', '--fastq_maxdiffs', '4', '--label_suffix', '_m1', '--fastq_qmax', '1000', '--threads', '2', '--fastq_allowmergestagger'], 'vsearch v2.0.3_linux_x86_64, 251.7GB RAM, 16 cores\nhttps://github.com/torognes/vsearch\n\n\n\nUnable to open file for reading (/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_an_266O.trimmedR1.fastq.tmp1)\n')
2017-12-29 19:00:39,852 pid=305993 [util.py] ERROR Error: ['/home/rjb/mfmazuecos/miniconda3/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '--fastq_mergepairs', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_or_287M.trimmedR1.fastq.tmp1', '--reverse', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_or_287M.trimmedR2.fastq.tmp2', '--fastqout', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/mar_or_287Mmerged.fastq', '--fastqout_notmerged_fwd', '/home/rjb/mfmazuecos/Helianthemum/data1_edits/tmpByROgn_nonmergedR1.fastq', '--fastqout_notmerged_rev', '/home/rjb/mfmazuecos/Helianthemum/dat

These are only a few of them. Could you tell me what could it be the problem? Something related to python version? Thanks you very much in advance!!
Deren Eaton
@dereneaton
Jan 02 2018 16:06
Hi @bioballs, (1) I found a bug in the baba code that was requiring way more RAM than it should, and which was probably making it freeze when you try to run many hundreds or thousands of tests at once. I'll push a new version very soon with an easy fix for it. (2) The bpp tree file is written to $HOME by default (a default of bpp, not ipyrad) and is a nexus file called FigTree.tre. I've been meaning to change this to have ipyrad grab that file and rename it based on the jobname and group it with the other output files (I'm not sure why that isn't the default behavior or bpp) but haven't done so yet. I'll take a look today and see if I can do that easily before pushing an update.
Isaac Overcast
@isaacovercast
Jan 02 2018 18:01
@Saritonia Happy new year! I'm willing to bet the problem is related to the parallel runs of the branches. Branches are guaranteed to operate independently if run independently (they dont interfere with one another). This isn't so true if run simultaneously, as some temp files are generated in such a way that two branches operating on them will almost certainly produce an error.
You can fix this by just running one branch at a time. There are other ways I could think to fix this so you could run two branches at the same time but they aren't so straightforward.