These are chat archives for ipython/ipython

11th
Sep 2015
Sylvain Corlay
@SylvainCorlay
Sep 11 2015 15:53
@jdfreder thanks for jupyter-devinstall
Remi Rampin
@remram44
Sep 11 2015 16:01
With multiprocessing, I can pass a Pipe or Queue object to the task (Process) I start, which allows me to communicate with it while it's running.
Can I do something similar when I submit a task to IPython.parallel instead?
Jonathan Frederic
@jdfreder
Sep 11 2015 16:14
@SylvainCorlay glad you liked it : D
If more people like it, I'd be happy to move it to a public org. Maybe it could be a guinea pig for the incubator process.
Sylvain Corlay
@SylvainCorlay
Sep 11 2015 16:17
Yes, it is a good idea
can always be in jovyans :)
Jonathan Frederic
@jdfreder
Sep 11 2015 16:33
I don't think I'm officially a jovyan : (
What's the process to become one?
May I become one?
Sylvain Corlay
@SylvainCorlay
Sep 11 2015 16:34
Just sent the invite!
Jonathan Frederic
@jdfreder
Sep 11 2015 16:44
Ok, repo is moving
Thanks
Min RK
@minrk
Sep 11 2015 16:46
@jdfreder neat tool. I still can't get over the desire to write filesystem scripting in async javascript.
Jason Grout
@jasongrout
Sep 11 2015 16:50
just wait until we write a filesystem in async javascript!
Min RK
@minrk
Sep 11 2015 16:51
You're killing me!
Jonathan Frederic
@jdfreder
Sep 11 2015 16:51
Probably wont be long till these guys/gals do it : D
Thanks : )
Min RK
@minrk
Sep 11 2015 16:52
Arg, nodeOS is the worst idea in the world!
Jason Grout
@jasongrout
Sep 11 2015 16:52
we essentially have a js filesystem with the contents api
Min RK
@minrk
Sep 11 2015 16:53
Yup, and you know what's better? Every single other way to work with a filesystem.
Jason Grout
@jasongrout
Sep 11 2015 16:53
hey, once they have a js filesystem, this can be a real computer: http://bellard.org/jslinux/
on the other hand, that probably already has a filesystem in js
seems like I can edit a file just fine
(that guy is definitely a hacker - wrote ffmpeg and qemu too...)
Min RK
@minrk
Sep 11 2015 17:02
Actually, if your target is deploying simple node-based webapps via docker, nodeos makes a lot of sense as a base image. Few resources, gets you straight to the stack you want, etc.
...If only node were good for writing web apps!
Jason Grout
@jasongrout
Sep 11 2015 17:06
sort of like the lisp machines, for lisp
Remi Rampin
@remram44
Sep 11 2015 17:06
my messages keep getting drowned :cry:
Min RK
@minrk
Sep 11 2015 17:12
@remram44 sorry. No, there is no Pipe/Queue for ipyparallel.
One could be built using sockets/files/etc. but there isn't one builtin.
What sort of thing do you want to do with it?
(this is a much more appropriate topic for this room than me being mean to javascript)
Remi Rampin
@remram44
Sep 11 2015 17:15
Things like asynchronously request stuff from the main process, or report progress
Min RK
@minrk
Sep 11 2015 17:16
You can get progress by publishing messages (display_data or data_pub)
Remi Rampin
@remram44
Sep 11 2015 17:19
that sounds very low level and is probably complicated to get right
Min RK
@minrk
Sep 11 2015 17:20
publishing messages isn't complicated - it's the same as displaying things in the notebook.
e.g. display(status)
Remi Rampin
@remram44
Sep 11 2015 17:21
getting that on the client is harder, and also it's just one direction
Min RK
@minrk
Sep 11 2015 17:21
Yeah, publishing status from the engiens to the client is the easy part. Interacting from the client to a running task is harder.
One way is to run the main task in a thread. We have an example using MPI
Remi Rampin
@remram44
Sep 11 2015 17:38
so do remote procedure calls with apply() for everything
Min RK
@minrk
Sep 11 2015 17:49
yup
Stefan van der Walt
@stefanv
Sep 11 2015 19:00
I have a message proxy written using pyzmq. It works perfectly fine by itself, but when run under Supervisord pushes up CPU usage to 100%. Any idea what might cause that?
Min RK
@minrk
Sep 11 2015 19:34
No, that's weird. Maybe something is signaling the process?
Can you run it for a bit with a profiler?
The most common thing that causes 100% CPU in a zmq app is a polling on an event and not handling it, so poll always returns immediately but then doesn't do anything.
@stefanv
Careful with learning-0mq-with-pyzmq, it's super outdated.
Jason Grout
@jasongrout
Sep 11 2015 19:39
pyzmq is super outdated?
Min RK
@minrk
Sep 11 2015 19:41
No, someone wrote a guide for learning about zeromq with pyzmq a long time ago, but it's hasn't been updated in years.
Min RK
@minrk
Sep 11 2015 19:48
@stefanv what's zmq.zmq_version()?
Stefan van der Walt
@stefanv
Sep 11 2015 20:05
@minrk Yes, that's the one!
Version 4.1.3
Min RK
@minrk
Sep 11 2015 20:05
ok, great. This may help: mltsp/message_flow#1
Stefan van der Walt
@stefanv
Sep 11 2015 20:06
Wow, @minrk. Thank you! Didn't expect a PR.
Where does a person learn the latest patterns?
Min RK
@minrk
Sep 11 2015 20:06
The zmq guide is typically pretty good.
The XSUB-XPUB proxy is what's described in the guide section you linked.
Stefan van der Walt
@stefanv
Sep 11 2015 20:08
Right. It's weird, though, that this works perfectly fine when run standalone, but only uses CPU under supervisord.
I should probably change to a ROUTER-DEALER setup soon.
Min RK
@minrk
Sep 11 2015 20:08
It could be an order of operations thing - that it only has a problem if the processes come up in a particular order.
Stefan van der Walt
@stefanv
Sep 11 2015 20:08
I will write up what I've done and get you guys to review it.
Min RK
@minrk
Sep 11 2015 20:08
Sounds good
Stefan van der Walt
@stefanv
Sep 11 2015 20:08
Are you using ROUTER-DEALER for IPython?
Min RK
@minrk
Sep 11 2015 20:08
(I'm not 100% sure that this will fix your issue, but it's possible)
Yeah, that's the main channel.
Stefan van der Walt
@stefanv
Sep 11 2015 20:09
I am using the message proxy to connect a bunch of flask servers to Tornado websocket servers.
Min RK
@minrk
Sep 11 2015 20:09
Multiple of each?
Stefan van der Walt
@stefanv
Sep 11 2015 20:09
Trying to get a uni-directional information flow (a la Flux) implemented in Python, as a template.
Yes, eventually. For now, just one Tornado server (with however many threads), and multiple Flask servers.
Min RK
@minrk
Sep 11 2015 20:10
If it's unidirectional, maybe use PUSH-PULL
ROUTER-DEALER is only necessary for sending replies to the right peer.
Stefan van der Walt
@stefanv
Sep 11 2015 20:11
Aha, thanks!
Min RK
@minrk
Sep 11 2015 20:11
PUSH-PULL is technically identical to DEALER-DEALER, as long as the flow is unidirectional.
ROUTER adds identification information of the peer each message comes from, needed to route replies. If you don't care about that, then the ROUTER can be PULL instead, and if you don't care about receiving replies, the DEALER can be PUSH.
Stefan van der Walt
@stefanv
Sep 11 2015 22:37
@minrk Your patch fixed the 100% cpu usage
Stefan van der Walt
@stefanv
Sep 11 2015 22:43
@minrk With PUSH-PULL, can you also have a subscription-like mechanism? I want to (eventually) only forward messages to nodes that need to know about them. Otherwise, I guess I should stick to PUB-SUB.