These are chat archives for dereneaton/ipyrad

20th
Jul 2017
James Clugston
@Cycadales_twitter
Jul 20 2017 00:13
@dereneaton yep followed above but with no luck at all. Any ideas?
Alexander McKelvy
@SnakeEvolution_twitter
Jul 20 2017 00:31
Hi guys, this might not be an iPyrad problem but I appreciate any insight you might have.
I am using SNAPP in Beauti/Beast. Some datasets (nexus files) load in okay (although slowly), and Beauti lets me export an XML file. For some imported files I can't export an xml. Details here: https://groups.google.com/forum/#!topic/beast-users/hWU61qhG-98
Could this be related to something iPyrad is doing in outputting the nexus files that I'm importing into Beast? I've manually inspected the beginnings and ends of the nexus files created by iPyrad for both nonworking and working examples, and they seem to be the same format, with no discernable pattern in number of taxa or characters to indicate sample or file size is causing the problem. These things make me think it's not an iPyrad problem, and I know this isn't Beast support, but if you can think of anything that might be causing this on the iPyrad front it could be very helpful in getting these stubborn datasets to work.
Thanks, Alex
joqb
@joqb
Jul 20 2017 07:50
Hi @dereneaton @isaacovercast ,
with a colleague we are trying to use a pop_assign_file. We have 69 individuals with both v.0.6.27 and v.0.7.2 with Deren's work around. Do you have any idea what might be the issue?
-bash-4.2$ ipyrad -p params-notean_nopoor.txt -s 7 -f -d

  ** Enabling debug mode ** 

 -------------------------------------------------------------
  ipyrad [v.0.7.2]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  loading Assembly: notean_nopoor
  from saved path: /data/filer-5-2/bernhardt/2017/GBS_wheat_group/assemblies_75_inds/notean_nopoor.json
  establishing parallel connection:
  host compute node: [32 cores] on bit-10.ipk-gatersleben.de

  Step 7: Filter and write output files for 69 Samples
  [####################] 100%  filtering loci        | 0:01:55  

  Encountered an error (see details in ./ipyrad_log.txt)
  Error summary is below -------------------------------
error in filter_stacks on chunk 0: IndexError(index 71 is out of bounds for axis 1 with size 69)
Nikki Phair
@phair_nikki_twitter
Jul 20 2017 11:11
Hi there, I have just installed ipyrad on the virtual machine, BioLinux, and I am attempting the intro tutorial but I am stuck creating the param file (>>> ipyrad -n iptest). I get... ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by /home/manager/Desktop/miniconda2/lib/libhts.so.1)
I have tried updating Ubuntu and searched for ways to manually update GLIBC but I am really a novice at this... Any advice @dereneaton ?
joqb
@joqb
Jul 20 2017 11:41

@phair_nikki_twitter Maybe you could try Deren's work around

Hi @fangbohao_twitter, this is a bug with the current installation which we're working quickly to try to fix. Right now it installs pysam v.0.6, whereas it should install pysam v.0.10 or higher. One way to work around this for now is to run the following conda commands in this order

conda remove ipyrad pysam samtools bcftools htslib 
conda install pysam=0.10.0 -c bioconda
conda install ipyrad -c ipyrad -f

This workaround will be temporary until we get the versions updated correctly so that they can all be installed from the ipyrad channel instead of requiring the bioconda channel.

Nikki Phair
@phair_nikki_twitter
Jul 20 2017 11:44
@joqb ,Thanks so much! I didn't see this. I will try it
Nikki Phair
@phair_nikki_twitter
Jul 20 2017 11:56
@joqb , this doesn't seem to help with the GLIBC version issue. Any other ideas?
joqb
@joqb
Jul 20 2017 12:09
@phair_nikki_twitter OK then have a look here http://ipyrad.readthedocs.io/faq.html#collisions-with-other-local-python-conda-installs or this http://ipyrad.readthedocs.io/installation.html#installation-troubleshooting-note I think in my case commenting out the miniconda path in my .bashrc and exporting it manually helped.
Nikki Phair
@phair_nikki_twitter
Jul 20 2017 13:26

@joqb unfortunately neither of those fixes help either. I'm still getting: manager@bl8vbox:~/Desktop/ipyrad_introtutorial$ ipyrad -n iptest
Traceback (most recent call last):
File "/home/manager/Desktop/miniconda2/bin/ipyrad", line 11, in <module>
load_entry_point('ipyrad==0.7.2', 'console_scripts', 'ipyrad')()
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 565, in load_entry_point

File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2598, in load_entry_point

File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2258, in load

File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/init.py", line 2264, in resolve

File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/init.py", line 20, in <module>
from . import load as _load
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/load/init.py", line 14, in <module>
from .load import test_assembly
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/load/load.py", line 13, in <module>
from ipyrad.assemble.util import
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/init.py", line 7, in <module>
from . import cluster_within
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/cluster_within.py", line 31, in <module>
from refmap import

File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/refmap.py", line 15, in <module>
import pysam
File "/home/manager/Desktop/miniconda2/lib/python2.7/site-packages/pysam/init.py", line 5, in <module>
from pysam.libchtslib import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by /home/manager/Desktop/miniconda2/lib/libhts.so.1)

Deren Eaton
@dereneaton
Jul 20 2017 16:09

Hey y'all, here's some more info on the GLIBC issue with the library pysam. If you run conda info you can see at the bottom of info it prints out, under the user-agent section, the version of the glibc compiler on your system. Conda does not install the C compiler, but rather uses the system compiler. On some systems (e.g., centos and red hat) the default compiler is sometimes older, and that is what's causing the problem. For example, on one system I have access to the glibc version is recent and there are no problems:

user-agent : conda/4.3.22 requests/2.12.4 CPython/2.7.13 Linux/4.4.0-81-generic debian/stretch/sid glibc/2.23

Whereas on another system I have access to the glibc is older and there are issues with the current pysam version in the ipyrad installation:

user-agent : conda/4.3.22 requests/2.14.2 CPython/2.7.13 Linux/3.10.0-327.el7.x86_64 Red Hat Enterprise Linux Server/7.3 glibc/2.17

Our current fix is to revert to an older version of pysam that is currently available from the bioconda channel that does not require glibc>2.17. To revert to the older version you must completely remove the currently installed versions, which can be done with the conda remove command:

conda remove ipyrad pysam samtools bcftools htslib
@jobq and @phair_nikki_twitter, can you run conda list and check that the version info for the following libraries are exactly like below.
bcftools                  1.3.1                         1    bioconda
htslib                    1.3.2                         0    bioconda
pysam                     0.10.0                   py27_3    bioconda
samtools                  1.3.1                         5    bioconda
Deren Eaton
@dereneaton
Jul 20 2017 16:24
Hey @SnakeEvolution_twitter, I don't think there is anything wrong with the NEXUS output from ipyrad. My guess is that the SNAPP package was not optimized for reading in large alignments because it probably expects to have very small alignments most of the time, since it is usually used just for analyzing SNPs, and because the runtime for an analysis of many thousands of SNPs would take a long time. You might be better off using the .snps.phy file, or even better, the .usnps.phy file from ipyrad, and then adding NEXUS headers to it.
Alexander McKelvy
@SnakeEvolution_twitter
Jul 20 2017 17:51
Thanks Deren, I will try that! I appreciate the suggestion very much!
Alexander McKelvy
@SnakeEvolution_twitter
Jul 20 2017 18:21
@dereneaton That appears to get me one step further for one of the datasets that wasn't working, hopefully it works for them all. Thanks again! SNAPP is supposed to be able to take an influx of data (and did okay in a few instances here) so I'm not sure why it couldn't parse the larger files, but I'm happy to get it working.
James Clugston
@Cycadales_twitter
Jul 20 2017 18:27
@dereneaton Hi Daren I am looking at what you are saying but I am unsure I fully understand. I have removed the packages but how do I install/ roll back the update. How do I install glibc>2.17 without root?
Deren Eaton
@dereneaton
Jul 20 2017 18:29
You cannot without root. Which is why we're trying roll back the pysam version to an older version which works with glibc 2.17.
James Clugston
@Cycadales_twitter
Jul 20 2017 18:30
Ahh ok. So how do I do that? I have tried the above commands without any luck or do you mean your trying to do this in an update to ipyrad?
Deren Eaton
@dereneaton
Jul 20 2017 18:34
After uninstalling everything you should be able to now install the new correctly versioned software by running conda install ipyrad -c ipyrad
BUT, the htslib will still not be correct (though I don't know why yet and it's driving me insane), so you then have to run conda install htslib=1.3.1 -c bioconda.
I'm trying to fix this stuff, but it's pretty complicated.
James Clugston
@Cycadales_twitter
Jul 20 2017 18:38
@dereneaton thanks sounds really annoying!
Deren Eaton
@dereneaton
Jul 20 2017 18:39
lol, yeah this is why we tried avoiding using any of these libraries for a long time, but they make the reference mapping steps much faster.
Deren Eaton
@dereneaton
Jul 20 2017 19:02
does that fix the problem for you? You can check quickly by opening an ipython session on the login node and typing import pysam.
James Clugston
@Cycadales_twitter
Jul 20 2017 20:00
@dereneaton no it did not help this time. I will try again
James Clugston
@Cycadales_twitter
Jul 20 2017 20:07
@dereneaton no still get the same error...well as everyone else is. Also this is not being run on a cluster but a server

```In [1]: import pysam

ImportError Traceback (most recent call last)

<ipython-input-1-9b87d3f66c91> in <module>()
----> 1 import pysam

/home/jclugston/miniconda2/lib/python2.7/site-packages/pysam/init.py in <module>()
3 import sysconfig
4
----> 5 from pysam.libchtslib import
6 from pysam.libcutils import

7 import pysam.libcutils as libcutils

ImportError: /home/jclugston/miniconda2/lib/././libcom_err.so.3: symbol k5_strerror_r, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference```

Isaac Overcast
@isaacovercast
Jul 20 2017 20:13
Im' working on this right now James
Isaac Overcast
@isaacovercast
Jul 20 2017 20:58
@Cycadales_twitter I can reproduce this error on one of my machines. I'll work on it.
Isaac Overcast
@isaacovercast
Jul 20 2017 21:06
@Cycadales_twitter You can fix this problem by doing this: export LD_LIBRARY_PATH=""
James Clugston
@Cycadales_twitter
Jul 20 2017 21:20
@isaacovercast Hi Isaac would you do that after you get the error or before? well I know you guys will fix it so I may just wait...but I do need to get the analysis finished!
Isaac Overcast
@isaacovercast
Jul 20 2017 21:22
Either is fine. This command will blank out the library search path, so the conda installed version of libkrb5 will be used rather than the local one. There's probably a more elegant solution to this, but i figured you'd rather get it running quick.
James Clugston
@Cycadales_twitter
Jul 20 2017 21:26
@isaacovercast Ahh ok it seems to be doing something now! well that was quite easy!
James Clugston
@Cycadales_twitter
Jul 20 2017 21:45

@isaacovercast I do have a question as I have had something strange happening for the past few weeks. I am running on a 96 core server in San Francisco and each time I run I am getting a range message with the number of core picked up. ---```----------------------------------------------------------
ipyrad [v.0.7.2]

Interactive assembly and analysis of RAD-seq data

New Assembly: Carm
establishing parallel connection:
host compute node: [1 cores] on tdobz.calacademy.org

Step 1: Loading sorted fastq data to Samples
[ ] 0% loading reads | 0:01:10 ```