These are chat archives for ipython/ipython

1st
Jul 2015
Bas Nijholt
@basnijholt
Jul 01 2015 12:25
should I open an issue for: :point_up: 29 juni 2015 17:47 ?
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 14:41
guys? anyone?
Steven Silvester
@blink1073
Jul 01 2015 14:41
still pretty early in Berkeley :smile:
This message was deleted
I’d say go ahead and file an issue on: https://github.com/jupyter/nbconvert
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 14:44
or in other words, if it's possible to create a kernel on a local machine that just mediates communication between a frontend (on the same local machine) and a kernel on a remote machine...
Steven Silvester
@blink1073
Jul 01 2015 14:44
@razvanc87, I was replying to @basnijholt’s question
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 14:46
@blink1073 I know, I wasn't asking you particularly, just in general.. people that might know... but I think one of my previous messages didn't get through
Steven Silvester
@blink1073
Jul 01 2015 14:46
I remember you had a question about hooking up two kernels, but I think it got lost in the shuffle
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 14:48
yep, so I'm reposting...
so the situation is this, there is this atom.io plugin, hydrogen that gives you real time interaction by running the code in the text editor just like the notebook does, but it does so by launching a local ipython kernel or connect to an existing one if there is one already around (I think)
but I would like to be able to connect this hydrogen thing to a remote kernel, and the way I thought about it is to add some custom kernelspec but I have no idea how to write all of this mediation... the documentation is really lacking... or at least I'm having a difficult time grasping it
I actually thought and hoped that you can connect a kernel to another kernel... as in ipython kernel --existing kernel-1234.json... and act as a communication layer... but you can't so I'm kind of stuck
Douglas La Rocca
@douglas-larocca
Jul 01 2015 14:53
hi, i've been following jupyter development pretty closely lately and I see this new phosphor library
will this be the future UI for the notebook?
Steven Silvester
@blink1073
Jul 01 2015 15:02
@douglas-larocca, right now it is a proof of concept, but we are working in that direction
@razvanc87, I think you want just one kernel, perhaps using this: https://github.com/ipython/ipython/wiki/Cookbook:-Connecting-to-a-remote-kernel-via-ssh
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:06
@blink1073 I already tried and know how to do that... but that only works when I want to connect an ipython frontend, such as console, or qtconsole. But the way this hydrogen thing works is... it looks for a running kernel on the local machine and if it doesn't find one it starts it then connects to the proper ports using zmq, so basically it uses a kernelspec file to start a kernel on the local machine if it doesn't find a running instance
hm... I need to try some things
Steven Silvester
@blink1073
Jul 01 2015 15:09
Personally, I would look at the source code to see what the —ssh switch is doing
(disclaimer: I’m not a Jupyter core dev)
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:10
yeah... well, that didn't help, first cause... it's a super mess that I don't understand, and second... the --ssh switch doesn't work for me, I need to do the manual port forwarding with the ssh command :)... so that's that
Steven Silvester
@blink1073
Jul 01 2015 15:12
well, if you set up the ssh forwarding, and change the kernel-*.json ip address to match that port...
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:13
yeah... that's what I want to do... but my lack of experience with ssh and zmq and coffeescript didn't help at all :))
Steven Silvester
@blink1073
Jul 01 2015 15:14
did you get a local ipython console working with the remote kernel?
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:15
yep, that's not a problem...
Steven Silvester
@blink1073
Jul 01 2015 15:16
then the problem is getting hydrogen to use the same kernel-json file that you used for the console
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:16
wait, I'm doing some gists now to better explain what I've been trying out...
Steven Silvester
@blink1073
Jul 01 2015 15:17
your best bet may be to post the question to Will here: https://github.com/willwhitney/hydrogen
@basnijholt, here’s a demo of an ipython console and rich display output using Phosphor
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:19

they're not so responsive it seems... :D, tried that already, my question is probably somewhere lost in... translation...

anyway, so I have this kernel.json file in $HOME/.local/share/jupyter/kenerls/fionn and I know I can run it with say ipython console --kernel fionn

Steven Silvester
@blink1073
Jul 01 2015 15:19
sidecar-browser.gif
Min RK
@minrk
Jul 01 2015 15:20
@razvanc87 you can open ssh tunnels to remote kernels. You shouldn't need an extra kernel to talk to the kernel.
I see that's discussed above, but that's the way to do it.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:22
@minrk ... so, again, the problem is that if I don't open a kernel that acts as a communication layer, then this hydrogen plugin instantiates a local kernel... so then I though of making this custom kernel that runs this fionnipk script to correctly set up the ssh port forwarding
I actually haven't tried forwarding the ports that the local kernel uses... to the remote host. The situation would be like this:
  1. I start remote kernel which gives me kernel-1234.json on the remote host
  2. I use hydrogen which starts a local kernel
  3. I forward the local kernels' ports to the remote ip correct ports?... <- this part is unclear to me
Min RK
@minrk
Jul 01 2015 15:33
You said you have already done ssh forwarding, right? So your kernel can just make that same ssh call.
So Hydrogen can't connect to existing kernels?
Matthias Bussonnier
@Carreau
Jul 01 2015 15:34
Hydrogen is still limited in kernel management.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:34
it can connect to existing kernels (I think), but local ones... this functionality is a bit lacking at the moment I guess
Matthias Bussonnier
@Carreau
Jul 01 2015 15:35
I think it still starts it's own kernel which is shared among docs.
Min RK
@minrk
Jul 01 2015 15:35
With ssh, the ports are local, so there's no difference between local and remote kernels.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:36
yes, but I think that hydrgoen looks for the process, if it doesn't find a local kernel the it starts it, that is the process... how would it know about a remote process?
anyway, yeah, I'm trying to get it working, if I do I'll post it somewhere...
Min RK
@minrk
Jul 01 2015 15:37
If hydrogen is looking for a process, it's doing it wrong. All it should ever need are ports.
But your idea about an intermediate kernel should work. All it should do is call out to ssh to make the tunnels.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:38
yes, this is what I'm trying to set up at the moment... unsuccessfully :)
rsmith31415
@rsmith31415
Jul 01 2015 15:42
@minrk Sorry to interrupt. Do you know why executing code right after interrupting the kernel generates an additional iopub message with response \n? This is what I mean:
from jupyter_client  import MultiKernelManager
from Queue import Empty
import threading
import time

km = MultiKernelManager()
id = km.start_kernel('python2')
kn = km.get_kernel(id)
client = kn.client()
client.start_channels()
client.wait_for_ready()

msg_id = client.execute('for i in range(1000000): print(i)')

def loop():
    response = ''
    print('start shell messages')
    while True:
        try:
            msg = client.get_shell_msg(timeout=5)
            print(msg)
        except Empty:
            continue

        if msg['parent_header']['msg_id'] == msg_id:
            break

    print('start iopub messages')

    while True:
        try:
            msg = client.get_iopub_msg(timeout=5)
            print(msg)
            if msg['msg_type'] == 'status':
                if msg['content']['execution_state'] == 'idle':
                    break
        except Empty:
            # execution state should return to idle before the queue becomes empty,
            # if it doesn't, something bad has happened
            raise

        content = msg['content']
        msg_type = msg['msg_type']

        if msg_type == 'execute_result':
            response = content['data']['text/plain']

        elif msg_type == 'error':
            response = content['ename']

        elif msg_type == 'stream':
            response = content['text']


    print('this is the response', response)
    return response


t = threading.Thread(target=loop)
t.start()

time.sleep(1)
kn.interrupt_kernel()
time.sleep(1)

msg_id = client.execute('1+1')
loop()
This outputs an iopub message with {content: {u'text': u'\n'}, ... } but this is intermittent.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 15:59
god! I have actually succeeded... (half way)... now I get 'Unsigned Message' error haha, but that's weird, I get this even with ipython kernel --no-secure...
Min RK
@minrk
Jul 01 2015 16:02
@rsmith31415 no, I don't know.
rsmith31415
@rsmith31415
Jul 01 2015 16:04
It only happens when executing something that prints. If I use for i in range(1000000): i, the output is correct every time.
This is the iopub message:
parent_header': {u'username': u'user', u'version': u'5.0', u'msg_type': u'execute_request', u'msg_id': u'a0b962b6-e95e-401a-9817-33965faea108', u'session': u'690a5f85-20e4-4d77-8fe1-aee577b8c9c9', u'date': datetime.datetime(2015, 7, 1, 10, 44, 46, 221594)}, 'msg_type': u'stream', 'msg_id': u'95e2f6d4-705f-4267-ab31-c12488b67d66', 'content': {u'text': u'\n', u'name': u'stdout'}, 'header': {u'username': u'user', u'version': u'5.0', u'msg_type': u'stream', u'msg_id': u'95e2f6d4-705f-4267-ab31-c12488b67d66', u'session': u'87cafd08-84c9-4056-a452-e479c819a492', u'date': datetime.datetime(2015, 7, 1, 10, 44, 46, 246677)}, 'buffers': [], 'metadata': {}
Min RK
@minrk
Jul 01 2015 16:11
That comes after the status-idle message?
rsmith31415
@rsmith31415
Jul 01 2015 16:12
Right before the idle message of the second execution.
Min RK
@minrk
Jul 01 2015 16:12
If so, it's likely a missing or mis-ordered sys.stdout.flush
rsmith31415
@rsmith31415
Jul 01 2015 16:13
Interestingly, I think the parent header points to the second execution, not to the interrupted loop.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 16:15
so.... I have finally did it!... I can connect this hydrogen with my remote kernel! don't know how but I did it :clap: for me... :)) I'll post the steps to make this work on a gist, hopefully someone else finds it useful... it took me 3 freaking days to understand all this...
rsmith31415
@rsmith31415
Jul 01 2015 16:16
Am I doing something wrong that triggers that last message?
(@razvanc87 :clap: )
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 16:17
thanks! :))
Min RK
@minrk
Jul 01 2015 16:42
@rsmith31415 if it's pointing to the subsequent exec, it's likely a missing flush
print actually makes two stdout.write calls - one for the line you give it, and another for the newline (or more generically, end)
rsmith31415
@rsmith31415
Jul 01 2015 16:44
Oh, that explains it.
Min RK
@minrk
Jul 01 2015 16:44
So if stdout.flush is being skipped, the last write may not get sent until the next flush call.
rsmith31415
@rsmith31415
Jul 01 2015 16:45
Should I do something in particular to prevent that from occurring?
Maybe running some command that triggers a flush before the second execute?
Min RK
@minrk
Jul 01 2015 16:51
If it's coming after the status-idle message, it's definitely a bug in the kernel.
@razvanc87 glad you figured it out. Sorry to not be more help.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 16:52
no worries, I'll give a gist about it shortly... for anyone else who might be interested
rsmith31415
@rsmith31415
Jul 01 2015 16:54
It is coming after the idle message of the first execute, but before the second idle message of the second execute. Is that a bug?
Min RK
@minrk
Jul 01 2015 17:03
Yes, I think so.
rsmith31415
@rsmith31415
Jul 01 2015 17:05
Okay. I will file a bug report. Thanks!
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 17:16
if anyone is interested... this is how I'm bending hydrogen to my will and make it interact with a remote kernel... though I guess it's kind of unsafe at the moment since we use the --no-secure option?... I'm unclear about this thing as I lack the proper knowledge in this field...
Steven Silvester
@blink1073
Jul 01 2015 17:17
Please update your issue on Hydrogen with a link to that gist, Will might want to add it to the readme
Min RK
@minrk
Jul 01 2015 17:18
thanks.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 17:19
I already did that... added the gist to the conversation that is
in the hydrogen tracker...
Steven Silvester
@blink1073
Jul 01 2015 17:20
gotcha, thanks, and well done @razvanc87!
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 17:22
yeah... thanks :) it turns out I don't understand anything from ipython's documentation hahahha which is kind of funny, only related through the fact that I sweat for putting together a communication layer type of kernel instead of port forwarding with ssh... until I discovered I was doing it all wrong... or well, that didn't get me anywhere
I think the documentation is lacking a bit... especially regarding the frontend-to-backend communication... I know and looked at that messaging passing page in the documentation, but it took me more than a day!! to figure out that you have to send information to the shell_socket using zmq.send_multi instead of zmq.send(json.dumps(...))!... lack of examples, that is :) anyway, I'm not complaining, you guys are doing very good work! I'm just saying that I was beginning to be frustrated :D
Min RK
@minrk
Jul 01 2015 17:30
@razvanc87 part of the reason for that is that if you are using Python, you probably shouldn't be making any zmq calls at all. the Session object in IPython should be used for all messages.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 19:25

@minrk... yeah, I've noticed some talk about that, but I couldn't find any examples (and the source code is beyond my understanding at this point...), it was just some conversation I saw somewhere... and it's a little bit difficult to start off, plus, the way I see it... is, this hydrogen plugin for example is working with zmq so you need to use zmq yourself in the "middle-layer", the situation is: frotend-with-zmq <-> (zmq-end | PythonManager | non-zmq-end) <-> kernel (possibly remote kernel)

so communication between frontend and the PythonManager is has to go through zmq, while at the other end you use only module provided by ipython... anyway, I did learn more things than I thought I would... so I'm happy about that at least :)

Min RK
@minrk
Jul 01 2015 19:31
The way I would have done it is to hook up the tcp sockets directly, and not used zmq in the middle layer at all (i.e. ssh tunnels).
@razvanc87 I understand there isn't much information for a use case like that, sorry about that.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 19:34
@minrk, yeah, no need to apologize, I think what's been done is quite amazing, I'm just saying that it's a bit difficult for newcomers to grasp what's going on, and my problem particularly was the lack of knowledge in networking protocols and such... which made things a bit more challenging (I mean I tried a lot of other different approaches before I got here haha)
Min RK
@minrk
Jul 01 2015 19:55
@razvanc87 not just newcomers, it's hard for us to keep track of it all, and we wrote it.
Răzvan Cosmin Rădulescu
@razvanc-r
Jul 01 2015 19:57
that doesn't sound too good... :), by the way I saw that little app, bipython... sadly it looks like a dead project, but it has some nice features... would some of those things eventually end up in ipython?
Jason Grout
@jasongrout
Jul 01 2015 21:05
Last week we were talking about a 4.0 release before scipy. How are we doing on that?
Min RK
@minrk
Jul 01 2015 21:16
We are going to have at least a few packages shipped, but not the whole shebang.
I think we'll have some time to sprint on the last bits, which are primarily documentation-related.
I don't think there's much code left to write for 4.0.
Jason Grout
@jasongrout
Jul 01 2015 21:18
If this ends up taking another month, I'd like to recast my vote in the message spec comm list question we had the other day :)
Min RK
@minrk
Jul 01 2015 21:19
I think we'll have most of it out by the end of SciPy sprints.
We should have core, nbformat, client, nbconvert, and maybe ipykernel before scipy.
and console,qtconsole, perhaps.
Mainly leaving ipython/ipython, notebook, and widgets as the pieces with some real polish work to do during SciPy.
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:32
thanks for your support @jasongrout
Do you guys know the folks who wrote nbdiff?
Jason was saying earlier than my ide is actually meld.
Jason Grout
@jasongrout
Jul 01 2015 21:34
Sylvain, I didn't say I'd change my vote :)
you sound like a politician :)
Min RK
@minrk
Jul 01 2015 21:36
@SylvainCorlay it was some students of Greg Wilson (Software Carpentry) in Toronto.
Fernando Perez
@fperez
Jul 01 2015 21:36
nbdiff was written by a group of undergraduate students in Canada, under the ...
never mind, @minrk types faster than me :)
Min RK
@minrk
Jul 01 2015 21:36
I think they've mostly moved on, but we were planning to pick it up soon, and see if we can bring it back up to speed.
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:38
well, could be a nice idea to look at a phosphor version of that :)
Min RK
@minrk
Jul 01 2015 21:38
sure.
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:39
ok, so it is abandoned. if we were to use some of the code, would be nice if it was BSD licensed
Min RK
@minrk
Jul 01 2015 21:39
as opposed to MIT?
Fernando Perez
@fperez
Jul 01 2015 21:39
it isn't???
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:39
it is MIT
I don't know if you can use MIT code in a BSD licensed software
Jason Grout
@jasongrout
Jul 01 2015 21:39
MIT is even more liberal than BSD
Fernando Perez
@fperez
Jul 01 2015 21:39
Basically the same thing.
Min RK
@minrk
Jul 01 2015 21:39
You definitely can.
Jason Grout
@jasongrout
Jul 01 2015 21:39
MIT > BSD > GPL
(in distributor freedom)
GPL > BSD > MIT (in end-user freedom)
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:40
ok
Jason Grout
@jasongrout
Jul 01 2015 21:40
with lots of overloaded meanings for freedom, etc.
Anton Akhmerov
@akhmerov
Jul 01 2015 21:44
FWIW I've used nbdiff quite extensively. Right now its merging abilities are limited to per-cell actions, which means a lot of follow-up manual work.
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:46
Thanks, now I have someone to ask question to ;)
Anton Akhmerov
@akhmerov
Jul 01 2015 21:55
Not sure if a better solution would be to use say something like http://json-delta.readthedocs.org/en/latest/ + adding metadata for cell diffs and a renderer for a diff notebook.
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 21:56
thanks for the pointer.
Anton Akhmerov
@akhmerov
Jul 01 2015 21:58
Are you going to be working on this?
Jason Grout
@jasongrout
Jul 01 2015 21:58
+1 to having a good diff algorithm for the notebook format
we'd need that for differential syncing, for example
diff and patch...
Sylvain Corlay
@SylvainCorlay
Jul 01 2015 22:00
Hey, by the way, I got back into this robotics stuff. Flight widgets + gamepad widgets :)
On the backend side, I started playing with sensors, but I need to do polling of values.
since I want to have a HasTraits interface to my sensors, I am plannig on spawning a new thread polling the sensors, which calls into the zmq event loop when I am in IPython to set the traitlets values
Kyle Kelley
@rgbkrk
Jul 01 2015 23:09
That's super neat