These are chat archives for dereneaton/ipyrad

27th
Nov 2017
J. Mark Porter
@jmark_porter_twitter
Nov 27 2017 04:40

Hi Deren, I've been running an ipyrad analysis on an HPC system. On recent runs I get the same error at different points in the analysis. Apparently what is happening is that at some point the assembly looks for a file but cannot find it:
ERROR:ipyrad.core.assembly:IOError(Unable to open file (Unable to open file: name = '/rhome/mporter/bigdata/ddrad/sclerob2_consens/15549_g_tmpcats.20955', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0))

However, the path defined in the parameter file is "/rhome/mporter/bigdata/ddRAD/" and the run is called "scleroB2" and the specimen name is "15549_G". It looks like all of the capitol letters are being changed to lowercase (not so good on a Unix machine). Do you think that this issue is being generated in ipyrad? Thanks for you thoughts.

Deren Eaton
@dereneaton
Nov 27 2017 18:38
Hi @jmark_porter_twitter, don't worry about the lower case lettering, that is just how the HDF5 library reports errors for some reason. The problem is that it is unable to open the designated file which most likely happened for one of two reasons, either you ran out of RAM or disk space. If it is a RAM issue then you can likely fix it by requesting more memory (we recommend 4GB of RAM per computing core), or by using fewer cores simultaneously. If the issue is disk space then you will need to request more of that as well. The temporary files in step5 can be large in some cases, which is something we're trying to improve.
Deren Eaton
@dereneaton
Nov 27 2017 18:45
Hi @Ollie_W_White_twitter, there are some options to modify the shape of the figure to make it easier to read but they are not documented yet, unfortunately. For now, you can increase the height of the canvas (height=1000), you can increase the width of the tree (pct_tree_x=0.75), and you can decrease the font size (style_test_labels={"font-size": "10px"}, style_results_label={"font-size": "10px"}, and style_tip_labels={"font-size": "10px"}). I hope to eventually improve the default options so that it fits better without needing to fiddle with it too much. That will wait for the next toytree update, though. ...
As for adding names to the results dataframe...
You can do the following to grab the names from the .tests attribute and add them as new columns to the the .results_table dataframe:
## initial results table

    dstat     bootmean     bootstd     Z     ABBA     BABA     nloci
0     0.148     0.148     0.030     4.910     532.781     395.750     9205
1     0.159     0.159     0.033     4.784     509.438     369.750     8981
2     0.134     0.135     0.033     4.060     492.938     376.688     8866
3     0.239     0.240     0.027     8.760     608.281     373.365     9266
4     0.290     0.290     0.031     9.247     606.312     333.812     8937
## initial tests dictionary
[{'p1': ['29154_superba'],
  'p2': ['41478_cyathophylloides', '41954_cyathophylloides'],
  'p3': ['33413_thamno'],
  'p4': ['32082_przewalskii', '33588_przewalskii']},
 {'p1': ['30686_cyathophylla'],
  'p2': ['41478_cyathophylloides', '41954_cyathophylloides'],
  'p3': ['33413_thamno'],
  'p4': ['32082_przewalskii', '33588_przewalskii']},
 {'p1': ['29154_superba'],
  'p2': ['41478_cyathophylloides'],
  'p3': ['33413_thamno'],
  'p4': ['32082_przewalskii', '33588_przewalskii']},
 {'p1': ['30686_cyathophylla'],
  'p2': ['41478_cyathophylloides'],
  'p3': ['33413_thamno'],
  'p4': ['32082_przewalskii', '33588_przewalskii']},
...
## code to add new columns to bb.results_table from the bb.tests
bb.results_table["p1"] = [i["p1"] for i in bb.tests]
bb.results_table["p2"] = [i["p2"] for i in bb.tests]
bb.results_table["p3"] = [i["p3"] for i in bb.tests]
bb.results_table["p4"] = [i["p4"] for i in bb.tests]
## new full results table (though a bit hard to read)
     dstat     bootmean     bootstd     Z     ABBA     BABA     nloci     p1     p2     p3     p4
0     -0.102     -0.102     0.043     2.371     275.438     338.312     6597     [29154_superba]     [41478_cyathophylloides, 41954_cyathophylloides]     [33413_thamno]     [32082_przewalskii, 33588_przewalskii]
1     -0.134     -0.133     0.038     3.534     316.125     413.875     8084     [30686_cyathophylla]     [41478_cyathophylloides, 41954_cyathophylloides]     [33413_thamno]     [32082_przewalskii, 33588_przewalskii]
2     -0.101     -0.098     0.043     2.327     269.375     329.938     6573     [29154_superba]     [41478_cyathophylloides]     [33413_thamno]     [32082_przewalskii, 33588_przewalskii]
3     -0.124     -0.121     0.038     3.284     312.188     400.250     8026     [30686_cyathophylla]     [41478_cyathophylloides]     [33413_thamno]     [32082_przewalskii, 33588_przewalskii]