These are chat archives for ipython/ipython

21st
Jun 2015
Douglas La Rocca
@douglas-larocca
Jun 21 2015 06:48
Hello
When migrating to jupyter, where should the files in profile_default/startup/ be put?
should these be broken down and made into extensions instead?
Douglas La Rocca
@douglas-larocca
Jun 21 2015 06:54
or specified in the config with c.InteractiveShellApp.exec_files
?
Douglas La Rocca
@douglas-larocca
Jun 21 2015 07:40
ok--so it seems to be working if I leave them in ~/.ipython/profile_default/startup
I suppose that's how it's intended?
Min RK
@minrk
Jun 21 2015 15:42
@douglas-larocca startup files are still part of IPython. They aren't going anywhere.
rsmith31415
@rsmith31415
Jun 21 2015 15:51
@minrk Quick question: What is the best way to implement kernel interruptions without the notebook. Obviously, interruptions are wanted when some code is running but this blocks additional requests to interrupt the execution. I tried using threads and that seems to work at first (there is an "Interrupting kernel ... " message) but doesn't actually interrupt the execution.
Min RK
@minrk
Jun 21 2015 15:53
I'm not sure what you mean by without the notebook. Are you writing your own app that uses kernels?
rsmith31415
@rsmith31415
Jun 21 2015 15:54
Exactly.
Min RK
@minrk
Jun 21 2015 15:54
Are you using the Python KernelManager API?
rsmith31415
@rsmith31415
Jun 21 2015 15:55
Yes. In particular, the MultiKernelManager.
Min RK
@minrk
Jun 21 2015 15:56
I believe there is a KM.interrupt (or interrupt_kernel) method
rsmith31415
@rsmith31415
Jun 21 2015 15:58
Yes, I tried that but the request is not received because of the current execution of code. I have seen a few implementations that digest the messages from the different channels (shell, iopub), but one of them involves a while loop and that could be blocking other requests.
Min RK
@minrk
Jun 21 2015 15:59
interrupt doesn't send a message, it sends a signal with os.kill
If the code is not responding to the signal it may be uninterruptible C code.
rsmith31415
@rsmith31415
Jun 21 2015 16:00
But you use a POST request to trigger the method to send the signal, correct?
Min RK
@minrk
Jun 21 2015 16:01
Yes and no. The browser sends POST to the notebook server, which then calls KM.interrupt.
is it your webserver that's stuck running code, not your kernel?
rsmith31415
@rsmith31415
Jun 21 2015 16:04
Sure. I don't think it is the code that is uninterruptible. It is a simple loop. I was wondering if the reason you're allowed to send a POST request is because you're using the ZMQ implementation of Tornado's websocket handler.
I believe so. The webserver seems to be the problem, although it is a bit confusing why using threads I'm able to send the interrupt signal, but the process keeps running.
Min RK
@minrk
Jun 21 2015 16:07
Blocking code in the kernel hasn't anything to do with the webserver's ability to handle POST requests, since the server is typically idle.
The kernel being busy and the server being busy are not generally related.
Are you using tornado for the parent application?
rsmith31415
@rsmith31415
Jun 21 2015 16:08
Correct.
Min RK
@minrk
Jun 21 2015 16:09
And are you using our existing kernel handlers?
Min RK
@minrk
Jun 21 2015 16:10
In your tornado Application, are you using those Handlers, or writing your own?
rsmith31415
@rsmith31415
Jun 21 2015 16:12
Well, I'm using a fairly basic workflow. Mainly starting kernels, using clients, but I'm not subclassing MainKernelHandler, IPythonHandler, etc because that looks related to specific utilities required by the notebook.
As an example, I have this:
if __name__ == '__main__':
    tornado.options.parse_command_line()
    app = Application()
    app.listen(8000)
    try:
        tornado.ioloop.IOLoop.instance().start()
    except:
        pass
    finally:
        tornado.ioloop.IOLoop.instance().stop()
        app.km.manager.shutdown_all()
Min RK
@minrk
Jun 21 2015 16:15
ok
rsmith31415
@rsmith31415
Jun 21 2015 16:15
app.km is a class that is creating kernels and other things. That includes the kernel manager.
Min RK
@minrk
Jun 21 2015 16:17
And you are using the same interrupt handler that we ship to turn POST into SIGINT?
rsmith31415
@rsmith31415
Jun 21 2015 16:18
Sure. the kn = km.get_kernel(id); kn.interrupt_kernel(); I also tried using the kernel manager directly: km.interrupt_kernel(id);
But as I said, the request is not being processed until the code execution finishes. That's why I thought that the while loop consuming the message (kc.get_iopub_msg) from the channel was a blocking function.
Min RK
@minrk
Jun 21 2015 16:21
get_iopub_msg is a blocking function.
Are you using BlockingKernelClient in your server?
rsmith31415
@rsmith31415
Jun 21 2015 16:24
I don't think so. I believe that is another class. When you use kn.client(), is that part of a blocking kernel client?
Min RK
@minrk
Jun 21 2015 16:24
To get messages in an async way that plays nice with tornado, you will probably want to use ZMQStream, like we do in the notebook.
rsmith31415
@rsmith31415
Jun 21 2015 16:25
Uhm, that was what I thought, but honestly I didn't want to because that adds another level of complexity.
ZMQStream, ZMQStreamHandler or AuthenticatedZMQStreamHandler?
Min RK
@minrk
Jun 21 2015 16:32
get_iopub_msg is a blocking recv. It won't return until a message arrives.
ZMQStream lets you register a handler for when messages arrive on a socket.
rsmith31415
@rsmith31415
Jun 21 2015 16:33
Min RK
@minrk
Jun 21 2015 16:35
Yes.
It's typically just two lines, given a socket:
stream = ZMQStream(socket)
stream.on_recv(my_handler)
Sylvain Corlay
@SylvainCorlay
Jun 21 2015 16:36
I made 3 more backport PRs. The most trivial although solving an annoying issue is #8556.
Min RK
@minrk
Jun 21 2015 16:36
And my_handler is a function that should be called when a message arrives on the socket.
rsmith31415
@rsmith31415
Jun 21 2015 16:37
Okay, I will take a look at it. Thanks. As a side note, if instead of returning a list in get_iopub_msg, you were to return a generator, then that would be non-blocking?
Min RK
@minrk
Jun 21 2015 16:38
@rsmith31415 then the first call wouldn't block, but asking for one item in the generator would, making it the same as before.
get_iopub_msg doesn't spend any time waiting between the first part and the last, so a generator doesn't help at all.
generators only help when generating each item takes some time, but get_iopub_msg only gets one item.
To make in non-blocking, you need to add a poller on the socket, and tell the eventloop to signal you when it's ready.
This can be done with callbacks (ZMQStream is an API that does exactly this) or coroutines.
rsmith31415
@rsmith31415
Jun 21 2015 16:41
But I was asking if that would help to use a coroutine in tornado in order to receive other requests without blocking using a callback or a timer.
Min RK
@minrk
Jun 21 2015 16:41
Sure, you could do that with a coroutine.
ZMQStream is that callback API.
rsmith31415
@rsmith31415
Jun 21 2015 16:43
I understand. Although I was referring to a tornado coroutine using a generator instead.
(you know, using add_callback or the yield Task pattern)
Min RK
@minrk
Jun 21 2015 16:45
sure, that would be fine.
rsmith31415
@rsmith31415
Jun 21 2015 16:45
But I can't because get_iopub_msg is a blocking function :-(
So the only option right now is ZMQStream to provide callbacks.
Min RK
@minrk
Jun 21 2015 16:47
get_iopub_msg(timeout=0)
Don't call it until you know there's a message to recv.
That's what your yield Task would need to do - wait for a message to be ready to recv.
All of get_iopub_msg's blocking is waiting for a message, so if you poll for that message in a tornado-async-friendly way (e.g. ZMQStream), it won't be blocking.
rsmith31415
@rsmith31415
Jun 21 2015 16:51
Oh, sure. However, I was asking hypothetically, what would happen if get_io_pub_msg returns a generator. Would I be able to do the same without ZMQStream? Because in that case, get_io_pub_msg would be non-blocking and coroutines in tornado would work as usual. Of course, that's a hypothetical.
Min RK
@minrk
Jun 21 2015 16:52
Just making it a generator doesn't help anything. It would have to be a tornado coroutine itself (a very specific kind of generator).
rsmith31415
@rsmith31415
Jun 21 2015 16:53
O a future?
Min RK
@minrk
Jun 21 2015 16:53
Sure.
The important thing is that the blocking step of the innermost generator (inside get_iopub_msg) must be making async tornado calls.
rsmith31415
@rsmith31415
Jun 21 2015 16:56
I was wondering because I didn't see other applications using the kernel API that implemented interrupt or shutdown. But maybe that is part of the reason. It is not as straightforward as just executing code in the client.
Min RK
@minrk
Jun 21 2015 16:57
You can make it non-blocking by waiting, as you said, with yield Task. If you don't call get_iopub_msg before there's a message for it to receive, it will not block.
That doesn't have to be in get_iopub_msg, it can be outside (making get_iopub_msg itself a generator is identical to wrapping it in one)
rsmith31415
@rsmith31415
Jun 21 2015 17:07
But without ZMQStream, I'm not sure if that works when a code execution is already blocking every request to the server. I need to think about it.
rsmith31415
@rsmith31415
Jun 21 2015 17:15
In any case,I think ZMQStream should be a better option because of the anomalies I found when interrupting the kernel using threads. interrupt_kernels gets called correctly but the code execution is still running. That's weird.