These are chat archives for ipython/ipython

29th
Jun 2015
Min RK
@minrk
Jun 29 2015 00:00
There shouldn't be any memory held in the server at all. Messages just pass through.
Also, there should only be one process identified as the 'notebook'. All the others should be some 'kernel' command, unless you are running several notebook servers.
epifanio
@epifanio
Jun 29 2015 00:01
i interrupt the kernel and now trying to load that specific notebook i got : Unreadable Notebook: /home/user/ipython/notebooks/OSGeo-live/osgeolive-gsoc-2015/notebooks/GDAL-OGR Quickstart.ipynb MemoryError()
Min RK
@minrk
Jun 29 2015 00:02
Once memory allocation stops working, you should expect ~everything to fail.
epifanio
@epifanio
Jun 29 2015 00:02
humm, gotcha. there should be only one notebook runing
htop is not so user friendly for debug, is there any command i can use to double check processing running ?
Min RK
@minrk
Jun 29 2015 00:03
You do need to be careful with htop - it will look like there are many processes, but it's actually reporting one line per thread of a single process.
Just top is typically simpler and clearer.
epifanio
@epifanio
Jun 29 2015 00:06
top show only one ipython process
using 60% mem
0% cpu
Min RK
@minrk
Jun 29 2015 00:17
And what's the libzmq version?
and Python version?
epifanio
@epifanio
Jun 29 2015 00:18
python 3.4.0 pyzmq 14.6.0
lubuntu 32 bit
on a vm using vmware
i’ll try to convert the notebook to a python kernel using “!” and using a different way to convert the image to png
and see if the problem is with the bash kernel + image magic and “display < ..."
Min RK
@minrk
Jun 29 2015 00:27
If it's just the large display, it should be reproducible by making a Python notebook with just one cell that displays the image you've already created.
epifanio
@epifanio
Jun 29 2015 00:40
i converted the notebook to python and avoid the use of image magic and “display <"
the memory grows but then get released
and I have no error messages
Min RK
@minrk
Jun 29 2015 00:45
interesting.
And it's definitely the server process whose memory grows?
And not the kernel's parent?
epifanio
@epifanio
Jun 29 2015 00:46
no stopping the kernel doesn’t free up memory
Min RK
@minrk
Jun 29 2015 00:46
The bash kernel is two processes: 1. bash itself, and 2. a Python process that handles the zmq stuff.
With top or ps you can see the command used to launch the process. Can you verify that it's the notebook server that's using lots of memory?
And definitely not the kernel process?
epifanio
@epifanio
Jun 29 2015 00:48
i’ll run the bash kernel from scratch and monitor the ps output
epifanio
@epifanio
Jun 29 2015 00:57
@minrk you want the output of ps all after/during the memory error happen, correct ?
Min RK
@minrk
Jun 29 2015 00:58
and before, ideally. And after shutting down the kernel (not restarting).
ps.log while was running at begin (no error)
ps3.log (same kernel running for the 3rd time) memory growing, no error
ps4.log same kernel, commands start to not respond correctly (0 displaied as output) error shows in the ipython shell log
ps5.log : output of ps all after “interrupting” the kernel (no restart)
Min RK
@minrk
Jun 29 2015 01:20
And what's the libzmq version?
zmq.zmq_version()
epifanio
@epifanio
Jun 29 2015 01:21
4.0.5
Min RK
@minrk
Jun 29 2015 01:21
We've had some other reports of apparent memory leaks that don't seem to have anything to do with zmq, only the notebook opening/saving.
I haven't been able to reproduce any of them, but I'll see what I can do based on this.
How big is the image?
epifanio
@epifanio
Jun 29 2015 01:23
i think that imagemagic command “convert” generate large jpg compared with gdal , up to 200mb
Min RK
@minrk
Jun 29 2015 01:23
So it's trying to display a 200mb image?
And it has the same problem when you try to display the same image in a Python notebook, with Image(filename='big.png')?
Do you have any configuration in your notebook server?
epifanio
@epifanio
Jun 29 2015 01:28
the image png or jpg in the bash kernel are 5, the biggest is 38mb
they are generated by converting a 300mb tif
that is not displayed
Min RK
@minrk
Jun 29 2015 01:29
Nevertheless, displaying the same image with the Python notebook using Image(filename=....) produces the same problem?
epifanio
@epifanio
Jun 29 2015 01:29
but for some reason … semms that goes into the memory
no the Image(filename=….) works just fine
Min RK
@minrk
Jun 29 2015 01:30
ok, so it's just the bash kernel that has this problem
epifanio
@epifanio
Jun 29 2015 01:30
yes
Min RK
@minrk
Jun 29 2015 01:30
And it's just the display call?
What is the single line that causes the memory to grow?
epifanio
@epifanio
Jun 29 2015 01:31
i don’t know if the “system call” to convert file.tif fire.png is also using memory
Min RK
@minrk
Jun 29 2015 01:31
Just calling convert causes the notebook server's memory to grow?
epifanio
@epifanio
Jun 29 2015 01:32
i’m reducing the example to few lines and check again
but seems so
Min RK
@minrk
Jun 29 2015 01:32
Try doing it one line at a time - one line per cell, and check memory after running each.
I need to go cook dinner, but I'll be back later to check in.
epifanio
@epifanio
Jun 29 2015 01:34
ok, buon appetito :)
i’ll try to reduce the issue to few lines and put the file.tif online
Min RK
@minrk
Jun 29 2015 01:35
The more detail you can come up with about which call is causing the memory to grow in which process, the better.
grazie
epifanio
@epifanio
Jun 29 2015 01:48
convert use 200 mb while performing the conversion and display use 100mb to render a 38.5 mb png
every other time i run conver
it first free up the memory (use 200mb, then release)
but waiting a little bit looking at the top output
the 200mb come back into memory
run several times the last 3 cell keeping a look at the memory usage
better wait a while between the execution of each cell to see which is the real process using the memory
for me … is “convert"
but i can be wrong
running the last 3 lines for say : 7 times use 1.5 gb mem
the converted jpg is in the order of 40 mb
Bas Nijholt
@basnijholt
Jun 29 2015 15:47
something in nbconvert broke...
try to convert to slides:
<p></p>
<h1 align=center>Title text<h1>
<p></p>
<h4 align=center>Bas Nijholt<h4/>

<center><img style="width: 500px" src="http://www.metasensing.com/wp/wp-content/uploads/2015/02/TU_d_line_P1_color_1.jpg"></center>
Screen Shot 2015-06-29 at 17.50.44.png
That's what it produces, while in live ipython notebook it looks OK.
Bas Nijholt
@basnijholt
Jun 29 2015 16:07
It used to work btw.
Also going back to IPython 3.1.0 doesn't help.
Min RK
@minrk
Jun 29 2015 16:38
paging @damianavila for slides question
Remi Rampin
@remram44
Jun 29 2015 22:32
Are these the docstrings for the modules? ^^ http://ipython.org/ipython-doc/dev/api/
Remi Rampin
@remram44
Jun 29 2015 22:40
How do I execute code in the kernel and get a result? e.g. how does the notebook do it?
Min RK
@minrk
Jun 29 2015 22:42
Yes, those are autogenerated docs from docstrings and signatures.
In the notebook, results are published as an execute_result message on the IOPub channel.
Remi Rampin
@remram44
Jun 29 2015 22:46
and what is that? I imagine that the _repr_*_ method will already have been called in the kernel and only that is serialized for the notebook server?
Min RK
@minrk
Jun 29 2015 23:01
Are you using the Python KernelClient API, or trying to run code from Javascript?
The publishing of the result is triggered when sys.displayhook is invoked at the end of execution.