These are chat archives for ipython/ipython

6th
Feb 2015
Brian E. Granger
@ellisonbg
Feb 06 2015 00:47
@takluyver does anyone (you?) have experience installing the R kernel on Windows?
How does one build rzmq?
Jonathan Frederic
@jdfreder
Feb 06 2015 00:50
@ellisonbg the quadcopter program uncovered some sort of bug in the widget framework. I can't really tell what's going on, but it's something along the lines of packets from the front-end to the back-end being dropped when there's a flood of packets going from the back-end to the front-end (which we don't throttle).
Brian E. Granger
@ellisonbg
Feb 06 2015 00:52
mmmm, very interseting - it is realted to the thread-safety?
Andreas Klostermann
@akloster
Feb 06 2015 00:55
I think I had a similar problem when trying to animate bokeh graphs .... there is a fundamental issue about throtteling data flow
and the problem will only get worse in remote situations
but it's nothing that the widget framework itself can solve
Andreas Klostermann
@akloster
Feb 06 2015 01:01
one solution is to keep the data slow from the kernel-side code, but that leaves a lot of performance and fluidness on the table, while probably still overwhelming slow browsers (think Raspberry Pi). But adaptive throttling is even harder, because bandwidth, latency and javascript processing speed goes into it
Jason Grout
@jasongrout
Feb 06 2015 14:17
HN comments could use some feedback about IPython/Jupyter: https://news.ycombinator.com/item?id=9004689
Jon Wilson
@jsw-fnal
Feb 06 2015 14:58
I have a question about jupyterhub security. I'd like to install jupyterhub (once there is a non-beta release) on a shared machine for our collaboration, and to open up a firewall hole for it. But, the sysadmin quite rightly asked me about security. How can I convince him that it is safe enough? (Is it safe enough, for that matter?)
Andreas Klostermann
@akloster
Feb 06 2015 15:47
no
Jon Wilson
@jsw-fnal
Feb 06 2015 15:49
^ that is in answer to "is it safe enough?" ?
Andreas Klostermann
@akloster
Feb 06 2015 15:50
in any useful form, Jupyter will always be a very fancy remote exploitation shell. Even if secured with a password, anyone who has the password/useraccount or someone who controls computers where a legitimate user logs in will have shell access to the machine running the kernels.
Jon Wilson
@jsw-fnal
Feb 06 2015 15:50
This machine also allows ssh logins with password.
Is it no more dangerous than that?
Andreas Klostermann
@akloster
Feb 06 2015 15:50
slightly more dangerous
Jon Wilson
@jsw-fnal
Feb 06 2015 15:51
what causes it to be slightly more dangerous, and how slight is slight?
Andreas Klostermann
@akloster
Feb 06 2015 15:51
because the security of the connections and login stuff aren't as thoroughly vetted
Jon Wilson
@jsw-fnal
Feb 06 2015 15:51
ok
Andreas Klostermann
@akloster
Feb 06 2015 15:51
you can tunnel through ssh
Jon Wilson
@jsw-fnal
Feb 06 2015 15:51
true, and that is what we are doing now, although it is a barrier to adoption
Andreas Klostermann
@akloster
Feb 06 2015 15:51
yes
Jon Wilson
@jsw-fnal
Feb 06 2015 15:52
but not a tremendous barrier
Andreas Klostermann
@akloster
Feb 06 2015 15:52
another solution is to run it on remote cloud servers you don't care about
Jon Wilson
@jsw-fnal
Feb 06 2015 15:52
haha, no good, our data is too big to put in the cloud at reasonable cost
Andreas Klostermann
@akloster
Feb 06 2015 15:52
I was afraid of that
Jon Wilson
@jsw-fnal
Feb 06 2015 15:53
and it may not be legal to do so in any case -- DOE-owned data living on non-DOE owned machines
Andreas Klostermann
@akloster
Feb 06 2015 15:53
I think your security people have to look at it more thoroughly
Jon Wilson
@jsw-fnal
Feb 06 2015 15:53
"security people" = one sysadmin
I never said it was a good setup. Just a legally and data-size encumbered setup...
Andreas Klostermann
@akloster
Feb 06 2015 15:54
I don't know how secure jupyterhub currently s
Jon Wilson
@jsw-fnal
Feb 06 2015 15:55
got it
but either way, putting it behind a firewall and requiring ssh tunnels makes it (approximately) no less secure than allowing ssh access already makes it
Andreas Klostermann
@akloster
Feb 06 2015 15:55
yes
Jon Wilson
@jsw-fnal
Feb 06 2015 15:56
that's probably good enough, then.
Andreas Klostermann
@akloster
Feb 06 2015 15:56
except that one could trick your users into running malicious notebook files ;-)
but I'm being paranoid now
Jon Wilson
@jsw-fnal
Feb 06 2015 15:56
It is a possibility
but we could be tricked into running malicious scripts as well.
perhaps less easily
Andreas Klostermann
@akloster
Feb 06 2015 15:58
security is hard
Jon Wilson
@jsw-fnal
Feb 06 2015 15:58
YES.
and unfortunately, it is also important.
Andreas Klostermann
@akloster
Feb 06 2015 15:59
and the more I learn about it the more I have to keep my mind from going all paranoid
Jon Wilson
@jsw-fnal
Feb 06 2015 15:59
one would like important things to be easy, and hard things to be unimportant.
but it is not usually the case.
Kyle Kelley
@rgbkrk
Feb 06 2015 16:26
@jsw-fnal As with any security, it's being evaluated continuously
There's been a lot we've locked down in the past releases
Especially with regards to opening malicious notebooks, hijacked kernels, etc.
For your own concerns at DOE, it would be really helpful if there was a way for the folks at Berkeley and Cal Poly to get funding dedicated to security and multiuser setups.
Through DARPA, Continuum has gone after the X-DATA research contracts
The nice bit is that at least it can be audited by anyone within your team
Matthias Bussonnier
@Carreau
Feb 06 2015 16:34
@minrk @takluyver , any chances you are already in barker hall ?
Rabi C Shah
@rabishah
Feb 06 2015 17:04
hey guys !
new to Python, so need some help. Actually, I want to create a Radio sort of server which streams music from Server. I tried it in node js but didnt get too far with it …
i am hoping for a quick headstart
Matthias Bussonnier
@Carreau
Feb 06 2015 17:12
Hi @rabishah
what did you looked at with Python ?
Rabi C Shah
@rabishah
Feb 06 2015 17:15
@Carreau i came across this CherryMusic
but i really have no idea as how to make a Radio station
Matthias Bussonnier
@Carreau
Feb 06 2015 17:16
It depends on your familiarity with python
I would not start with radio streaming, I woudl start with somethign simple.
typically a FlaskApp
Rabi C Shah
@rabishah
Feb 06 2015 17:18
thanks !
Matthias Bussonnier
@Carreau
Feb 06 2015 17:19
If you want to do some web;
Rabi C Shah
@rabishah
Feb 06 2015 17:19
i’ve used socket io and all
Matthias Bussonnier
@Carreau
Feb 06 2015 17:19
also be aware the IPython is good for exploratory work, probably not that good to write a server.
So you are more a JS guy.
Rabi C Shah
@rabishah
Feb 06 2015 17:20
@Carreau ha ha .. exactly
but i really wanna do this personal project of streaming songs just for my friends at home
so i realized nodejs is not the solution
then i came across Python
Matthias Bussonnier
@Carreau
Feb 06 2015 17:20
Well python is a bit different than JS, I would start with flask, to get something easy, then move toward Tornado
Flask is good, but mostly blocking.
tornado is more complex, but you can do websocket and more asynchronous things.
If you have the choice start with python >= 3.3
all the async stuff is much better in 3 than in 2.7
Rabi C Shah
@rabishah
Feb 06 2015 17:22
which i dont mind,
hmmmm … but it looks like it will take a long time to get familiar with these things
Matthias Bussonnier
@Carreau
Feb 06 2015 17:23
Not that much. I leaner the flask basics in 2 days.
It's really nice for quick and dirty.
Rabi C Shah
@rabishah
Feb 06 2015 17:23
okay, do you have any idea as how i can make a streaming server (or underlying principles) behind it ?
not in terms of Python itself … in general
Matthias Bussonnier
@Carreau
Feb 06 2015 17:24
I guess tornado with websocket would be pretty nice if you need to "push" data.
if your MP3 are already on the server and relatively small, I wouldn't bother streaming and just push the all MP3 in an audio player.
Rabi C Shah
@rabishah
Feb 06 2015 17:26
shouldn’t it be done like bytes wise ?
instead of sending the whole file
then, you are actually sending the same bytes to all the clients
Matthias Bussonnier
@Carreau
Feb 06 2015 17:26
bytes per bytes, maybe not, but by chuncks I guess.
there are question of wether you allow clients to seek or not.
Jonathan Frederic
@jdfreder
Feb 06 2015 17:28
Did someone say bites? :hamburger:
Good morning @Carreau , how's the first few days in the states?
Rabi C Shah
@rabishah
Feb 06 2015 17:28
client should not be able to seek basically ,
Matthias Bussonnier
@Carreau
Feb 06 2015 17:30
@jdfreder second day actually :-)
Jonathan Frederic
@jdfreder
Feb 06 2015 17:31
first couple* days
;)
Matthias Bussonnier
@Carreau
Feb 06 2015 17:31
ah, sorry.
@rabishah I woudl search websocket and html5 api to stream audio.
so you woudl probably directly need to use tornado for that.
Jonathan Frederic
@jdfreder
Feb 06 2015 17:32
No you are right, few = 3, couple = 2. Do you have an apartment?
Rabi C Shah
@rabishah
Feb 06 2015 17:33
@Carreau thanks man !
its not related to iPython but I will try it out
Andreas Klostermann
@akloster
Feb 06 2015 18:10
there are open source internet radio servers and the easiest way to build a web server hosting a radio stream would be to install such a streamer on the server and then a simple web app to stream the audio in an html5 audio component. (I think src="url" is all the networking needed on the js/html side)
Andreas Klostermann
@akloster
Feb 06 2015 18:30
Is there any issue with ipython/jupyter I could help with?
Matthias Bussonnier
@Carreau
Feb 06 2015 18:42
@akloster some issues shoudl be tagged sprint friendly.
though we are in feature-freeze because of Beta, so testing PR tagged 3.0 and writing docs are probably the only things we will do during the next week(s)
Jon Wilson
@jsw-fnal
Feb 06 2015 19:50
@rgbkrk , thanks. The computer(s) in question are actually at Stanford (although I am not). Maybe I can send the admin (who is only adminning part-time, he's also a physicist) over to chat with you guys if he is unsatisfied with any arrangement I can come up with.
where you guys ≡ the Berkeley IPython folks
Kyle Kelley
@rgbkrk
Feb 06 2015 19:51
Ah, well then. An admin came over to talk to us when we had our spring dev meeting last year. They're looking to integrate with Stanford's OAuth setup and run users in namespaced docker containers
Jon Wilson
@jsw-fnal
Feb 06 2015 19:52
not the same, I think
Kyle Kelley
@rgbkrk
Feb 06 2015 19:52
I'm guessing not yours, but it's worth saying others are going to be doing it too
Jon Wilson
@jsw-fnal
Feb 06 2015 19:52
Indeed so
and it sounds like that guy has a lot bigger fish than our little tiny handful of machines
(I think our "cluster" is 3 machines in the Stanford Underground Facility)
Min RK
@minrk
Feb 06 2015 19:53
The folks who talked to us were in neuroimaging
Jon Wilson
@jsw-fnal
Feb 06 2015 19:53
ok
Kyle Kelley
@rgbkrk
Feb 06 2015 19:53
ah, thanks @minrk
Jon Wilson
@jsw-fnal
Feb 06 2015 19:54
we're in dark matter, and our computing is scattered across a number of universities and national labs, but most data analysis is done on the SUF cluster
Jon Wilson
@jsw-fnal
Feb 06 2015 19:59
Next question: If I request a number of nodes from PBS to run IPython.parallel engines, the nodes may not all become available at the same time. Will the client pick up the additional nodes as they come online?
("client" means IPython.parallel.Client)
The task I'm currently thinking of seems to match reasonably well with load_balanced_view and map (although I don't need the result pickled and sent back -- is there an alternative that is more like for than map?)
so ideally, I would start the cluster and immediately give some tasks to the lview. Then as engines came online or finished a previous task, they would each pick up the next available task until they were all done.
Related: can I programmatically shut down the ipcluster when all the tasks are finished? This would keep me from holding dozens of nodes idle after things are done.
Min RK
@minrk
Feb 06 2015 20:10
@jsw-fnal yes, it will pick them up as they arrive
There isn't an equivalent to for, but you can wrap the function in a lambda that returns None before passing it to map
e.g. map(lambda f, *a, **kw: f(*a, **kw) or None, real_f, *real_args, **real_kwargs)
Jonathan Frederic
@jdfreder
Feb 06 2015 20:14
@jasongrout @SylvainCorlay I found a nasty bug in the widget stuff that may affect you.
My theory is that it's related to the promises...
But I haven't tested it yet.
Min RK
@minrk
Feb 06 2015 20:15
I promise* it's related
* there may be a bug in this promise
Jonathan Frederic
@jdfreder
Feb 06 2015 20:15
LOL
When sending a flood of messages from the back-end to the front-end, it's possible to "break" the front-end so it can't send messages back.
I'll open an issue and upload the notebook that reproduces it.
Min RK
@minrk
Feb 06 2015 20:16
I think the original plan was to cut an rc / b2 today, but I think we should work a bit more on the output widget stuff and multi-select in the tree view first, so maybe eary next week instead.
Jonathan Frederic
@jdfreder
Feb 06 2015 20:17
Did you see my PR to revert the output widget stuff?
Min RK
@minrk
Feb 06 2015 20:18
I did, thanks. I think that's the right way to go for 3.0.
Jon Wilson
@jsw-fnal
Feb 06 2015 20:18
@minrk merci beaucoup, that helps. parallel is a slick tool.
Jonathan Frederic
@jdfreder
Feb 06 2015 20:29
Wait, nvm, doh
I see why this is failing
hmm
well there is a bug, this is just a bad example notebook.
Min RK
@minrk
Feb 06 2015 20:31
That's my favorite kind of bug!
^ that replicates it
After a couple seconds of the pseudo thread running and the user simultaneously dragging the slider, a flood of zmq related garbage appears in the terminal.
:(
    ValueError: '<IDS|MSG>' is not in list
[C 12:39:36.042 NotebookApp] Malformed message:
and stuff like
[C 12:39:36.043 NotebookApp] Malformed message: ['{}', '{"execution_state":"idle"}']
    Traceback (most recent call last):
      File "/Users/jon/ipython/IPython/html/base/zmqhandlers.py", line 166, in _on_zmq_reply
        msg = self._reserialize_reply(msg_list, channel=channel)
      File "/Users/jon/ipython/IPython/html/base/zmqhandlers.py", line 149, in _reserialize_reply
        idents, msg_list = self.session.feed_identities(msg_list)
      File "/Users/jon/ipython/IPython/kernel/zmq/session.py", line 756, in feed_identities
        idx = msg_list.index(DELIM)
    ValueError: '<IDS|MSG>' is not in list
Min RK
@minrk
Feb 06 2015 20:46
@jdfreder you can't use the zmq sockets at all from threads
we can add locks to prevent this sort of thing
Jonathan Frederic
@jdfreder
Feb 06 2015 20:47
Oh that would be a good patch
One of the students in this lab
was trying to interface with Crazyflie
and the mechanism that polls data from it
runs on another thread
so naively setting the widget values to the values pulled in
occasionally causes a crash
although I'm not sure that that's the only thing gone wrong.
He polling program seems to work fine
if it's just sending data to the front-end
Min RK
@minrk
Feb 06 2015 20:49
makes sense. IOLoop.add_callback is the only way to do anything that sends or receives data from another thread.
Jonathan Frederic
@jdfreder
Feb 06 2015 20:49
ah ok
hmmm
Min RK
@minrk
Feb 06 2015 20:51
it hands off the call to the main threqd
Jonathan Frederic
@jdfreder
Feb 06 2015 20:51
ok
when he gets back
I'll have him look at his console
to see if he is seeing the same threading error message.
Min RK
@minrk
Feb 06 2015 20:52
we can put a check in comms that do this if called not from the main thread.
Jonathan Frederic
@jdfreder
Feb 06 2015 20:52
If he is seeing that, I'm satisfied. :P
mmm but all of the IOLoop callback registering happens pre-widgets
so I don't see how that would help his particular issue
since that the Crazyflie thread is unrelated to the thread that instantiates the widgets/comms etc.
When I verify his bug, I'll mark the issue I opened as a duplicate of the "widgets aren't thread safe" issue which is still floating around somewhere.
Jason Grout
@jasongrout
Feb 06 2015 21:01
@jdfreder - we got a widget to crash a browser consistently
of course, it was sending megabytes of data to javascript...
Jonathan Frederic
@jdfreder
Feb 06 2015 21:01
:P
Jason Grout
@jasongrout
Feb 06 2015 21:01
still, it was really annoying to have widget persistence, because we couldn't ever load the page
Sylvain Corlay
@SylvainCorlay
Feb 06 2015 21:02
We should create the CrashWidget
Jonathan Frederic
@jdfreder
Feb 06 2015 21:04
He's not sending more than 10kB/s though, just a bunch of small messages (6*10 messages a second, pitch, yaw, roll, and an acceleration vector).
And the front-end only stores the most recent state (6*4byte floats).
Sylvain Corlay
@SylvainCorlay
Feb 06 2015 21:04
a crash widget is really necessary for a flight widget collection
as long as drones will fly, they will fall
Jonathan Frederic
@jdfreder
Feb 06 2015 21:05
@SylvainCorlay what do you mean crash widget? :stuck_out_tongue: One that intentionally crashes the browser?
Sylvain Corlay
@SylvainCorlay
Feb 06 2015 21:05
yes
Jonathan Frederic
@jdfreder
Feb 06 2015 21:06
lol
Jason Grout
@jasongrout
Feb 06 2015 21:09
Seriously, it would be great if there were a way to recover from bad data in localstorage from widgets
bad = too much in this case
it locks up the browser while the javascript tries to load and create widgets
Sylvain Corlay
@SylvainCorlay
Feb 06 2015 21:12
  • Clear all output could also clear the widget states from local storage.
  • And be accessible from the dashboard
Jonathan Frederic
@jdfreder
Feb 06 2015 21:13
@minrk he was seeing the x-threading error
indeed
Jonathan Frederic
@jdfreder
Feb 06 2015 21:35
@SylvainCorlay @jasongrout so that script would need to run either before the persistence, or on the dashboard?
Maybe like a "Clear widget cache" button?
Jason Grout
@jasongrout
Feb 06 2015 21:36
clear widget cache sounds good.
maybe it can be an option that shows up if you check the notebook name in the dashboard
though it seems like it doesn't deserve a whole button's worth of space
maybe when you check notebooks, it could be a "More actions" dropdown
Jonathan Frederic
@jdfreder
Feb 06 2015 21:39
Regardless, the localStorage persistence is experimental for 3.0. An MVP of sorts. When we have a file based persistence, it will become irrelevant. I remember talking about adding a flag that allowed you to prevent a widget from being persisted, but I don't think we actually implemented that.
Jason Grout
@jasongrout
Feb 06 2015 21:40
the localstorage option is really nice for refreshing a page talking to a running kernel
Jonathan Frederic
@jdfreder
Feb 06 2015 21:40
True
So maybe we'll need both?
Well, but the notebook data comes from the server on page refresh
Nothing says the widget data couldn't too
So I don't think we'd need both
the file persistence would be enough.
Jason Grout
@jasongrout
Feb 06 2015 21:41
that works too
Jonathan Frederic
@jdfreder
Feb 06 2015 21:42
Or not-file based server side storage
like in memory.
Jonathan Frederic
@jdfreder
Feb 06 2015 22:26
@minrk I'm talking to @ellisonbg , do you think it's okay to set the xthread bug to 3.0 or 3.1?
Instead of 4.0 ?
I don't understand how large the fix you suggest would be, so I don't know if this is a sane thing to do.
Min RK
@minrk
Feb 06 2015 23:01
what xthread bug?
It's been true since the beginning of two-process IPython, so I wouldn't block 3.0 with it
Min RK
@minrk
Feb 06 2015 23:08
I can fix it for Comms at least with a few lines
Min RK
@minrk
Feb 06 2015 23:14
@jdfreder #7706
We may want to make an IOLoop-aware Session object that does this on all sends. That would make it harder to do the wrong thing, but I think that would be a bigger fix than we should try to get into 3.0.
Jonathan Frederic
@jdfreder
Feb 06 2015 23:24
Thanks @minrk ! That looks great. Now I understand what you were suggesting