mirnylab/cooler

Nezar Abdennur

@nvictus

hello

Nezar Abdennur

@nvictus

This message was deleted

switching rooms because the old name sucked

nandankita

@nandankita

Hi, good morning!

Is there any problem in your Jupyter notebook?

http://mybinder.org/repo/mirnylab/cooler-binder

It fails to load.

nandankita

@nandankita

Also, I got this error today while creating .cool file. Will you please let me know what does it mean?

chr6_ssto_hap7
Traceback (most recent call last):
File "/usr/bin/cooler", line 9, in <module>
load_entry_point('cooler==0.5.2', 'console_scripts', 'cooler')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 716, in call
return self.main(args, kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(args, **kwargs)
File "/home/ankita/4DN/cooler/cooler/cooler/cli/cload.py", line 83, in cload
create(h5, chroms, lengths, bins, reader, metadata, assembly)
File "/home/ankita/4DN/cooler/cooler/cooler/io/init.py", line 66, in create
bin1_offset, nnz = write_pixels(grp, n_bins, reader, h5opts)
File "/home/ankita/4DN/cooler/cooler/cooler/io/_writer.py", line 157, in write_pixels
for chunk in reader:
File "/home/ankita/4DN/cooler/cooler/cooler/io/_reader.py", line 209, in iter
for chunk in self._iterchunks(chrom):
File "/home/ankita/4DN/cooler/cooler/cooler/io/_reader.py", line 187, in _iterchunks
for record in pairsfile.fetch(chrom, bin.start, bin.end, parser=parser):
File "pysam/ctabix.pyx", line 455, in pysam.ctabix.TabixFile.fetch (pysam/ctabix.c:5292)
ValueError: could not create iterator for region 'chr6_ssto_hap7:1-10000'

Nezar Abdennur

@nvictus

Hi Ankita!

It fails to load.

The binder service is not super stable. Sometimes it takes a few tries. Sometimes the service is down temporarily.

Seems like the scaffold chr6_ssto_hap7 wasn't available in your contact data, but was in your chromsizes file. If that's true, this is a bug. The workaround for now would be to limit your chromsizes file (and the bins file generated from it) to only those contigs that were mapped to in your data.

Indeed, the Binder service is currently down: http://mybinder.org/status

nandankita

@nandankita

Thanks Nezar!! It works now, and yeah that was the issue as you mentioned about my chromsizes file. I didn't know, I just downloaded hg19.chrom.sizes file from ucsc. Anyway Thanks a lot!

nandankita

@nandankita

Sorry Nezar, but I ran into another issue while running this command:
cooler dump --header --balanced HB_Cool.cool | head

bin1_id bin2_id count balanced
1 2059 1
1 8659 1
1 9335 1
5 9247 1
6 3015 1
6 4647 1
7 3307 1
8 1278 1
8 2774 1
Traceback (most recent call last):
File "/usr/bin/cooler", line 9, in <module>
load_entry_point('cooler==0.5.2', 'console_scripts', 'cooler')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 716, in call
return self.main(args, kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(args, **kwargs)
File "/home/ankita/4DN/cooler/cooler/cooler/cli/dump.py", line 162, in dump
f, sep='\t', index=False, header=False, float_format='%g')
File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 1344, in to_csv
formatter.save()
File "/usr/lib64/python2.7/site-packages/pandas/formats/format.py", line 1551, in save
self._save()
File "/usr/lib64/python2.7/site-packages/pandas/formats/format.py", line 1652, in _save
self._save_chunk(start_i, end_i)
File "/usr/lib64/python2.7/site-packages/pandas/formats/format.py", line 1678, in _save_chunk
lib.write_csv_rows(self.data, ix, self.nlevels, self.cols, self.writer)
File "pandas/lib.pyx", line 1075, in pandas.lib.write_csv_rows (pandas/lib.c:19767)
IOError: [Errno 32] Broken pipe

Please let me know what am I doing wrong.

Nezar Abdennur

@nvictus

Hey Ankita! You're doing nothing wrong, actually. You can essentially ignore that message: it's printed to stderr, so if you redirect the output it won't carry over. It's caused by head closing the pipe, and cooler should be suppressing the error, but I think it's not getting caught in Python 2. Let me try patching it.

I pushed a patch. Try pulling changes from master and let me know if the traceback goes away.

nandankita

@nandankita

Ohh Thanks! I will just try that now.

nandankita

@nandankita

Yeah the patch fixed the error! To do this step:
mat = c.matrix(balance=True).fetch('chr5:10,000,000-15,000,000')
I need to have a .cool file which has bins/weight column,
So, first I thought if I just do balancing, it should be good, but that doesn't seem to be the case, should I calculate weight first and then write to the file?

Nezar Abdennur

@nvictus

Yes, if you use cooler balance it will write the weights to the file.

nandankita

@nandankita

Do you mean this is correct?
cooler dump --header --balanced HB_Cool.cool | head
bin1_id bin2_id count balanced
1 2059 1
1 8659 1
1 9335 1
5 9247 1
6 3015 1
6 4647 1
7 3307 1
8 1278 1
8 2774 1

of course I did this first
cooler balance HB_Cool.cool
variance is 115532.067933
variance is 1880.1202993
variance is 142.980174133
variance is 16.5192880813
variance is 1.92663361978
variance is 0.25196914841
variance is 0.0346924669294
variance is 0.00558787607212
variance is 0.00103577668573
variance is 0.000235077586937
variance is 6.00085555451e-05
variance is 1.68418189549e-05
variance is 4.89101936518e-06

Nezar Abdennur

@nvictus

yes... if a bin was filtered during balancing, it will have a weight of NaN. So the balanced value will be NaN, which prints as empty

you can check with cooler dump --annotate weight

nandankita

@nandankita

Nezar, I tried do cooler balance and then cooler dump, then I thought weights should be there in my cool file, but I still get this error through python API
matb = c.matrix(balance=True).fetch('chr5:10,000,000-15,000,000')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "cooler/core.py", line 360, in fetch
File "cooler/api.py", line 311, in _slice
File "cooler/api.py", line 560, in matrix
ValueError: No column 'bins/weight' found. Use cooler.ice to calculate balancing weights.

Is there any step I am missing?

Also another thing, I tried is cooler show, which again gave me error :(

cooler show --out test.png --dpi 200 HB_Cool.cool chr3:0-80,000,000
Traceback (most recent call last):
File "/usr/bin/cooler", line 9, in <module>
load_entry_point('cooler==0.5.2', 'console_scripts', 'cooler')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 716, in call
return self.main(args, kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(args, kwargs)
File "/home/ankita/4DN/cooler/cooler/cooler/cli/show.py", line 135, in show
'log10' : 'log 10 ( relative contact frequency '}[scale]))
File "/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py", line 1976, in colorbar
ret = gcf().colorbar(mappable, cax = cax, ax=ax, kw)
File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1387, in colorbar
cb = cbar.colorbar_factory(cax, mappable, kw)
File "/usr/lib64/python2.7/site-packages/matplotlib/colorbar.py", line 1173, in colorbar_factory
cb = Colorbar(cax, mappable, kwargs)
File "/usr/lib64/python2.7/site-packages/matplotlib/colorbar.py", line 854, in init
ColorbarBase.init(self, ax, **kw)
TypeError: init() got an unexpected keyword argument 'label'

Am I missing steps?

So sorry to bother you like this. I am just running into the errors!

Nezar Abdennur

@nvictus

No worries. These reports are helpful!

The first one is surprising

If you restart the python interpreter, is the weights column still missing?

You can also check using h5py.

f = h5py.File('myfile.cool', 'r')
'weight' in f['bins']

Anton will look into cooler show. I suspect it's related to the matplotlib version. Could you print the results of pip list?

nandankita

@nandankita

I am home and can't able to connect to my system from here. I'll let you know first thing tomorrow morning!

Anton Goloborodko

@golobor

I think I fix this issue in the master branch. Nezar, how long ago did you merge develop back into master? The master branch somehow didn't get the proper imports from matplotlib (i.e. using "Agg" when out is None)

Nezar Abdennur

@nvictus

yeah, that change is in develop... i'll cherry-pick it over to master

nandankita

@nandankita

[ankita@localhost cooler]$ pip list
-ipap11helper (0.1)
backports.ssl-match-hostname (3.4.0.2)
Beaker (1.5.4)
biopython (1.68)
blivet (0.61.15.37)
Brlapi (0.6.0)
cffi (0.8.6)
chardet (2.2.1)
click (6.6)
configobj (4.7.2)
configshell-fb (1.1.18)
cooler (0.5.2, /home/ankita/4DN/cooler/cooler)
coverage (3.6b3)
cryptography (0.8.2)
cupshelpers (1.0)
decorator (3.4.0)
di (0.3)
dnspython (1.12.0)
enum34 (1.0.4)
ethtool (0.8)
firstboot (19.5)
freeipa (2.0.0a0)
fros (1.0)
fuse-python (0.2.1)
h5py (2.6.0)
iniparse (0.4)
initial-setup (0.3.9.30)
ipaplatform (4.2.0)
ipapython (4.2.0)
IPy (0.75)
javapackages (1.0.0)
kerberos (1.1)
kitchen (1.1.1)
kmod (0.1)
langtable (0.0.31)
lxml (3.2.1)
Mako (0.8.1)
MarkupSafe (0.11)
matplotlib (1.2.0)
netaddr (0.7.5)
nose (1.3.0)
ntplib (0.3.2)
numpy (1.11.1)
pandas (0.18.1)
Paste (1.7.5.1)
perf (0.1)
pip (7.1.0)
ply (3.4)
policycoreutils-default-encoding (0.1)
pyasn1 (0.1.6)
pycparser (2.14)
pycups (1.9.63)
pycurl (7.19.0)
pyfaidx (0.4.6)
pygobject (3.14.0)
pygpgme (0.3)
pyinotify (0.9.4)
pykickstart (1.99.66.6)
pyliblzma (0.5.3)
pyOpenSSL (0.13.1)
pyparsing (1.5.6)
pyparted (3.9)
pysam (0.9.1.4)
pysmbc (1.0.13)
python-augeas (0.5.0)
python-dateutil (1.5)
python-default-encoding (0.1)
python-dmidecode (3.10.13)
python-ldap (2.4.15)
python-meh (0.25.2)
python-nss (0.16.0)
python-yubico (1.2.3)
pytz (2012d)
pyudev (0.15)
pyusb (1.0.0b1)
pyxattr (0.5.1)
qrcode (5.0.1)
rtslib-fb (2.1.57)
scipy (0.18.0)
seobject (0.1)
sepolicy (1.1)
setroubleshoot (1.1)
setuptools (0.9.8)
six (1.9.0)
slip (0.4.0)
slip.dbus (0.4.0)
SSSDConfig (1.13.0)
targetcli-fb (2.1.fb41)
Tempita (0.5.1)
urlgrabber (3.10)
urwid (1.1.1)
yum-langpacks (0.4.2)
yum-metadata-parser (1.1.4)
[ankita@localhost cooler]$

nandankita

@nandankita

After restarting python interpreter it worked for balance=True ! Also, now the 'weight' in f['bins'] is True as well.

nandankita

@nandankita

Can I try cooler show now?

nandankita

@nandankita

I am just wondering, if there is any way to create a cool file which is not binned?

Anton Goloborodko

@golobor

yes, you can try cooler show with the current version on master

btw, what do you mean by a cooler file that's not binned? Is it a file with individual read pairs? What kind of analyses do you have in mind?

nandankita

@nandankita

So, I am not sure if it is correct to ask but we do these steps
$ cooler makebins $CHROMSIZES_FILE 10000 > bins.10kb.bed
$ cooler cload bins.10kb.bed $CONTACTS_FILE out.cool

As I understood I am getting a cool file which is 10 kb binned. What if I don't want it to be binned and create a cool file ? Does that make sense?

Where communities thrive

People

Repo info

Activity