These are chat archives for dereneaton/ipyrad

30th
Mar 2017
James Clugston
@Cycadales_twitter
Mar 30 2017 15:27
@isaacovercast @dereneaton Hi guys any ideas about this error? 2017-03-22 10:43:30,768 pid=103844 [assembly.py] ERROR IPyradWarningExit: error in singlecat (Cmac-15-10-2-R_S19001_PairedTrim) EngineError(Engine '8433a984-8db6-421a-aab8-9c543f3dd8f4' died while running task u'8a8e78a6-57bd-4330-b44c-2d24903924e1') 2017-03-23 02:49:40,911 pid=11746 [assembly.py] ERROR float division by zero 2017-03-23 10:50:14,450 pid=12029 [assembly.py] ERROR IPyradWarningExit: error in singlecat (Cmac-15-10-5-R_S22001_PairedTrim) EngineError(Engine '8ba98f0e-4040-4004-bf4f-217a8026f79d' died while running task u'1dbc4c66-cb48-43ca-91df-c6f67d7c49e5') 2017-03-30 08:21:10,030 pid=34956 [assembly.py] ERROR IPyradWarningExit: error in singlecat (Cmac-15-10-4-R_S21001_PairedTrim) EngineError(Engine 'a897fd41-44cf-4680-8a56-b1ae4f454d93' died while running task u'907eb021-2c3c-4d9b-8f76-8c6d04f13221') 2017-03-30 08:24:42,079 pid=108320 [load.py] WARNING Load assembly error setting params. Not critical b/c new params file may correct the problem. Recorded here for debugging: Error setting parameter 'output_formats' 'tuple' object has no attribute 'strip' You entered: ('p', 's', 'u', 'n', 'k', 'g', 'v')
Deren Eaton
@dereneaton
Mar 30 2017 16:31

@Cycadales_twitter , yes, this was a bug that was introduced recently. A hotfix for it right now is to go into your JSON file and find the part labeled "output formats" and replace the value (which will be a dictionary) with the string "psunkgv". Make sure there is a comma after it, and that you use double quotations, not single, like below:

"output_formats": "psunkgv",

This bug is fixed for the next release.

@Cycadales_twitter , Oh, I see you have a different error occurring actually. The other one is just a warning.
@Cycadales_twitter there is a float division by zero happening in step 6 it seems.
James Clugston
@Cycadales_twitter
Mar 30 2017 17:20
@dereneaton yes that happened during step six during indexing clusters. What is a float division and how do I correct it? is it just the one sample? or did this one sample make the other not work. I have actually rerun from step three a few times with the dataset now with the same issue.
Deren Eaton
@dereneaton
Mar 30 2017 17:29
It means there is an empty array occurring somewhere where we expect there to be data. Is there anything unusual about that sample (very high or very low amounts of data?)
James Clugston
@Cycadales_twitter
Mar 30 2017 17:43
@dereneaton it does not look like it...I have attached the step three stats final stats and its does not looks like anything is wrong to me.
I have also attached the final stats which did not work due to the error but it does state number of reads etc.
Jenny Archibald
@jenarch
Mar 30 2017 17:50
Hi @isaacovercast or @dereneaton, Unfortunately, a week later and with more nodes this time - my analysis still isn't moving off of 0% clustering for Step 6. Also, the separate analysis of the same data that I started from the beginning (in case the many restarts with different ipyrad versions had messed things up) failed with many errors in step 3. Most of the errors said something about not finding files:
"
2017-03-29 14:13:43,122 pid=36917 [cluster_within.py] ERROR sample [Ls_BRU_9fr] failed in step [build_clusters]; error: ValueError(too many values to unpack)
2017-03-29 19:22:46,157 pid=36917 [cluster_within.py] ERROR sample [Qm_HEB_19] failed in step [muscle_align]; error: IOError([Errno 2] No such file or directory: '/panfs/pfs.local/scratch/bi/jkarch/cam/ch2/m04c90-tmpalign/Qm_HEB_19_chunk_6.ali')
2017-03-29 19:22:46,164 pid=36917 [cluster_within.py] ERROR sample [Ls_PHR_2M] failed in step [muscle_align]; error: IOError([Errno 2] No such file or directory: '/panfs/pfs.local/scratch/bi/jkarch/cam/ch2/m04c90-tmpalign/Ls_PHR_2M_chunk_9.ali')
...
2017-03-29 19:47:18,811 pid=36917 [jointestimate.py] ERROR Sample Ls_MDF_V failed with error ValueError(cannot convert float NaN to integer)
2017-03-29 19:47:18,858 pid=36917 [jointestimate.py] ERROR Sample CA_OTT_2M failed with error ValueError(cannot convert float NaN to integer)
2017-03-29 19:47:19,476 pid=36917 [assembly.py] ERROR The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
"
Can you give any suggestions?
Deren Eaton
@dereneaton
Mar 30 2017 18:15
@jenarch Hi Jenny, have you moved some of your files, or is it possible that they got removed from scratch automatically? It looks like the file /panfs/pfs.local/scratch/bi/jkarch/cam/ch2/m04c90-tmpalign/Qm_HEB_19_chunk_6.ali is expected to be there but it is missing.
Jenny Archibald
@jenarch
Mar 30 2017 18:17
@dereneaton They aren't supposed to be moved by anyone else for at least 6 months, and I don't see how I could have done it on accident. I can ask the cluster people about it though.
Deren Eaton
@dereneaton
Mar 30 2017 18:18
@jenarch as for the slow clustering problem. If it is still running then there should be a temporary file that is created called {projdir}/{name}_consens/{name}_catshuf.tmp. Do you see that file? And if so can you send it to me?
Jenny Archibald
@jenarch
Mar 30 2017 18:20
@dereneaton It hit wall time limits and so is not actively running right this second, but I do see that file. Should I email it to you?
Deren Eaton
@dereneaton
Mar 30 2017 18:21
Jenny Archibald
@jenarch
Mar 30 2017 18:33
It is over 1.5 G, is that typical? Because of the size, I'm going to email you a dropbox link to the file once it syncs. Thanks for your help!
Deren Eaton
@dereneaton
Mar 30 2017 18:36
Yes, it it contains all of the consensus reads from all of your samples, so it should be big. A link is fine.
Deren Eaton
@dereneaton
Mar 30 2017 18:51
@jenarch can you send me your params file as well
James Clugston
@Cycadales_twitter
Mar 30 2017 19:00
@dereneaton do you have any suggestions on how to correct the issue I am having as I am not sure its sample related.