These are chat archives for ipython/ipython

26th
Nov 2014
Kyle Kelley
@rgbkrk
Nov 26 2014 07:19
Anyone still awake?
Min RK
@minrk
Nov 26 2014 07:31
What's up @rgbkrk?
Kyle Kelley
@rgbkrk
Nov 26 2014 07:32
Nonsense from me mostly, wanted to show something off that we talked about a while ago.
Takes a bit to load there
but eh
Only need it work every now and then
I'm going to shuttle the images off to an object store/CDN anyway
Wanted to experiment with what our Twitter/G+/Whatever cards looked like
went ahead and clobbered some node libraries together (including Phantom like you all had mentioned)
Not much code here since phantom+webshot+restify do all the heavy lifting
Min RK
@minrk
Nov 26 2014 07:36
doesn't load quite right on my phone
Kyle Kelley
@rgbkrk
Nov 26 2014 07:37
:(
well
if its loading the base path, its going to look ugly
Min RK
@minrk
Nov 26 2014 07:38
oh, I get it
Kyle Kelley
@rgbkrk
Nov 26 2014 07:38
because our root path on nbviewer looks ridiculous at low resolution
Min RK
@minrk
Nov 26 2014 07:38
neat
Kyle Kelley
@rgbkrk
Nov 26 2014 07:38
Its just making images at the right size for social media
Min RK
@minrk
Nov 26 2014 07:39
I thought it was rendering a page wrong, I didn't realize it was an image.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:40
yeah
whoopsy me not giving context
clearly shouldnt be coding at 0141
oh well, I'll keep going with the rest of this
Min RK
@minrk
Nov 26 2014 07:43
That's cool. You can work on hilarious heuristics for finding the "most interesting" area to screenshot.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:43
yeah, I was thinking first image
Maybe make this jupyter/nbimage, iterate on it there
its yet another not specific to ipython/jupyter project
its also been done a bunch, but I want the stream to go straight to a CDN and get built in mind for our needs on nbviewer
Let's all pretend that I know what I'm doing in JavaScript.
Min RK
@minrk
Nov 26 2014 07:45
If you made it specific to nbviewer, you could target around the first or last figure, or heading, or some such.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:45
Yeah... hmmm
it should be specific to nbviewer
Min RK
@minrk
Nov 26 2014 07:45
lol, nobody knows what they are doing in Javascript.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:46
because when searching around, there were dozens of screenshot tools
Min RK
@minrk
Nov 26 2014 07:46
You could do similar heuristics for generic pages, but reasoning about a notebook is a lot easier than reasoning about webpages in general.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:46
If we get this working we can actually rotate that front page out cleanly
Min RK
@minrk
Nov 26 2014 07:47
You could make it pluggable, so the web slinging part is generic, but deployments can specify the heuristics for how to decide what region to screenshot.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:47
ah, hmmm
Not sure of the best way to make the heuristics pluggable
Min RK
@minrk
Nov 26 2014 07:48
Everything else is generic, right?
Kyle Kelley
@rgbkrk
Nov 26 2014 07:48
except for making this a library
but this is like 42 lines of code (without error checking that is...)
yeah
Everything else is generic
Can you tee a stream in node?
Like, send a readable stream to two writable streams?
Only for testing here
Min RK
@minrk
Nov 26 2014 07:49
probably, and probably not with the stdlib. You can't do anything with the node stdlib.
Kyle Kelley
@rgbkrk
Nov 26 2014 07:49
ok
I'll just send on to CDN now in a branch, see how it goes
Min RK
@minrk
Nov 26 2014 07:51
Well, it's probably easy, actually. With stream.on("data") or something. Just register more than one callback. But I bet it isn't already implemented in the stdlib.
yuppppp
Min RK
@minrk
Nov 26 2014 08:01
awesome. That's the one right there.
Kyle Kelley
@rgbkrk
Nov 26 2014 08:02
lol
I'm noticing inconsistency from the rendered image
I'm guessing you see the current, which looks a bit janky
Kyle Kelley
@rgbkrk
Nov 26 2014 08:09
@minrk do you think if I close the stream with the client it ends any other background work on the event loop?
since I don't have a real API here I'm returning res.send({}); now
I get an almost immediate success from writeStream.on('success'...
Kyle Kelley
@rgbkrk
Nov 26 2014 08:25
aww durr
it was working fine
i forgot that cloudfiles now "browses" paths
Jason Grout
@jasongrout
Nov 26 2014 18:24
@jdfreder?
Jonathan Frederic
@jdfreder
Nov 26 2014 18:25
hi
I looked at ipython/ipython#6990 earlier this morning, it looks good right now.
I was thinking about it some more
I think it makes sense to utils.reject(...,true) for all top level promise methods (including events) AND public facing methods (even if they aren't top level).
but I don't think it makes sense to log things that are private and are always used by another logging method
since the wrapped error maintains a stack, and it will all get logged anyways
in ipython/ipython#6990 it looks like that's the same conclusion as the one you came to
Jason Grout
@jasongrout
Nov 26 2014 18:30
I just made every reject have true as the last argument. That's why I thought it should be true by default. I'm not sure what you mean by " log things that are private and are always used by another logging method"
I'm looking through the widget persistence. It seems that you load widget state when the callbacks are set. Why? Usually the first time you set the callbacks, the widget state hasn't been saved yet.
Jonathan Frederic
@jdfreder
Nov 26 2014 18:31
on page refresh
Don't " log things that are private and are always used by another logging method" -
I'll write a quick example snippet, one second...
The following code does not make sense if _b is only called internally by another logging promise method:
var _c = Promise.resolve(-1);

var _b = function() {
    return _c
        .then(function() {return 0;})
        .catch(utils.reject('b failed', true));
};

var a = function() {
    return _b()
        .then(function() {return 1;})
        .catch(utils.reject('a failed', true));
};
Jonathan Frederic
@jdfreder
Nov 26 2014 18:36
Instead the following is better:
var _c = Promise.resolve(-1);

var _b = function() {
    return _c
        .then(function() {return 0;})
        .catch(utils.reject('b failed'));
};

var a = function() {
    return _b()
        .then(function() {return 1;})
        .catch(utils.reject('a failed', true));
};
Jason Grout
@jasongrout
Nov 26 2014 18:37
okay, sure. It's not a huge deal to me.
(to have a default true).
Jonathan Frederic
@jdfreder
Nov 26 2014 18:38
Nor is it to me, I was just clarifying my reasoning
Jason Grout
@jasongrout
Nov 26 2014 18:38
sounds great.
Jason Grout
@jasongrout
Nov 26 2014 18:43
Looking through the persistence PR, I don't see an easy way to just store the currently active and visible models + the javascript link models.
Perhaps we could pass a whitelist of model types/modules into get_state as an option
Jonathan Frederic
@jdfreder
Nov 26 2014 18:46
I'd be okay adding a whitelist argument, but why would you want to persist Javascript links but not other invisible models?
Jason Grout
@jasongrout
Nov 26 2014 18:47
I suppose there are lots of other active widgets that were part of old cells. The views got deleted, so the model now is not visible.
a=IntSlider()
then realize that oops, you wanted it to be b, so you change it to b=IntSlider().
a is still around. I'd rather not persist it.
but the javascript link widgets are things that intentionally don't have views.
Jonathan Frederic
@jdfreder
Nov 26 2014 18:49
Ah I see
I think that would be easy enough to add
I could make it work by instance, id, type, or a combination of the three.
Jason Grout
@jasongrout
Nov 26 2014 18:51
yes. I'm still looking for a better solution to the general problem of links
Jonathan Frederic
@jdfreder
Nov 26 2014 18:51
whitelist = get_visible(manager._models).concat(JavascriptLink)
Jason Grout
@jasongrout
Nov 26 2014 18:51
at this point, maybe it's better to just invoke a callback to filter the list.
Jonathan Frederic
@jdfreder
Nov 26 2014 18:52
that's a good idea
Jason Grout
@jasongrout
Nov 26 2014 18:52
So maybe you could leave it as it is, but have a new option that is a callback to make a decision.
Jonathan Frederic
@jdfreder
Nov 26 2014 18:52
If I do that, I could remove the other options
Jason Grout
@jasongrout
Nov 26 2014 18:52
I think the other options are probably still good. They are very common cases, I would think
Scott Sanderson
@ssanderson
Nov 26 2014 19:01
@rgbkrk I swear you're reading the quantopian hive mind with your jupyter projects...was about to start doing research for how to do social sharing of notebooks and saw that you'd pushed up https://github.com/jupyter/nbshot/commits/master last night...
Kyle Kelley
@rgbkrk
Nov 26 2014 19:06
Ha!
well that's just for screenshots to be used by nbviewer for Twitter cards and the trending notebooks page
Scott Sanderson
@ssanderson
Nov 26 2014 19:08
we're thinking about two pieces: one is twitter cards and facebook and all that jazz
and the other is embeddable widgets for blogs and whatnot
Kyle Kelley
@rgbkrk
Nov 26 2014 19:08
easiest way to share right now is posting to github or a host of your own, then posting to nbviewer
"easiest"
we should have a notebook store you can publish to, IMO
Scott Sanderson
@ssanderson
Nov 26 2014 19:09
that's what we're going to end up building for ourselves I think
Kyle Kelley
@rgbkrk
Nov 26 2014 19:09
ah yeah the embeds able widgets would be good
do you think you'll open source that?
Scott Sanderson
@ssanderson
Nov 26 2014 19:11
unclear; I was in the stage of "see what exists in terms of prior art" when I ran into nbshot
Kyle Kelley
@rgbkrk
Nov 26 2014 19:11
the notebook store is on my plate as is indexing notebooks, etc
Scott Sanderson
@ssanderson
Nov 26 2014 19:11
parts of what we build are going to be pretty tightly integrated with our existing forums and sharing infrastructure
Kyle Kelley
@rgbkrk
Nov 26 2014 19:11
the next fun pieces for nbshot will be creating heuristics for what's interesting on the page
ah, k
Scott Sanderson
@ssanderson
Nov 26 2014 19:11
but I could imagine a world where the backend infrastructure would be shareable
it's also related to the project of persisting the notebooks that are stored in jupyterhub/tmpnb containers
Kyle Kelley
@rgbkrk
Nov 26 2014 19:12
Yeah
Scott Sanderson
@ssanderson
Nov 26 2014 19:12
b/c however those notebooks are stored is going to also be where we're pulling from for sharing
Kyle Kelley
@rgbkrk
Nov 26 2014 19:13
part of me is waiting to integrate with the Jupyter Google drive project
Scott Sanderson
@ssanderson
Nov 26 2014 19:14
yeah, that's tricky for us b/c we're storing people's sensitive IP, so we probably don't want to be relying on a third party service
Jason Grout
@jasongrout
Nov 26 2014 19:14
@ssanderson: we implemented embeddable widgets in the sage cell project. People have been embedding them for years (not ipython widgets, though, because those rely on bootstrap and other things that take over a page)
(we = me and a few Drake University students)
Scott Sanderson
@ssanderson
Nov 26 2014 19:15
@jasongrout do you have an example you could point me to of one of those widgets embedded in another site?
Jason Grout
@jasongrout
Nov 26 2014 19:15
Go to http://sagecell.sagemath.org/, click "About" in the upper right, then scroll down to "where is it used" for some examples
Scott Sanderson
@ssanderson
Nov 26 2014 19:15
also @rgbkrk, w/r/t heuristics for choosing interesting data, our plan was to let sharers explicitly choose a cell to highlight
Kyle Kelley
@rgbkrk
Nov 26 2014 19:16
Yeah I've thought about that
and I'd like to enable that
but at the same time it should work when they don't do that
Scott Sanderson
@ssanderson
Nov 26 2014 19:17
for us the workflow of sharing will probably involve picking a cell to highlight
do you imagine a sharing case where that wouldn't make sense?
Kyle Kelley
@rgbkrk
Nov 26 2014 19:17
weve also talked about standards for embedding/including style sheets. That hasn't really progressed
no that makes sense
just sometimes people want to share a whole workflow
Scott Sanderson
@ssanderson
Nov 26 2014 19:18
right, I think you should be able to drill down from the preview to get back to the full nb
but if you're designing some digest format for social media/embedding, a cell seems like a pretty natural unit for that
Kyle Kelley
@rgbkrk
Nov 26 2014 19:19
ah yeah
embedding is an interesting bit
wonder how far we can push twitter
or any other platform, like facebook
Jason Grout
@jasongrout
Nov 26 2014 19:20
even if embedding doesn't work, we'd share links that resolved to filled and executing sage cells
Kyle Kelley
@rgbkrk
Nov 26 2014 19:20
guessing they won't let us run JavaScript though
ah, cool
Jason Grout
@jasongrout
Nov 26 2014 19:21
so that's what I did on Google+, for example.
Scott Sanderson
@ssanderson
Nov 26 2014 19:21
@jasongrout where are the servers hosted that are backending those running cells?
I'm assuming those cells have to be self-contained?
the servers are VMs at University of Washington, on the Sage math project infrastructure
Scott Sanderson
@ssanderson
Nov 26 2014 19:22
gotcha
Jason Grout
@jasongrout
Nov 26 2014 21:43
@jdfreder, @minrk - is there a big reason why comm messages are send-and-forget, and don't have the notion of replies? The underlying protocol has the notion of replies to a certain message. Is it to simplify things, or is it so we can run comm messages over lighter weight protocols, or something?
Jason Grout
@jasongrout
Nov 26 2014 21:52
For example, one of jdfreder's widget persistence things is a call/reply to get the state from the server side over a comm message. He has to go through some awkwardness to send the state and then say "yeah, that last message was a response".
Min RK
@minrk
Nov 26 2014 22:23
The decision was largely made for symmetry. There is no mechanism for the kernel to request/reply from the frontend, only the other way around.
So if Comms are a symmetrical protocol, there can be no request-reply
Jonathan Frederic
@jdfreder
Nov 26 2014 22:24
We could always build request/reply on top of comms, if needed...
Min RK
@minrk
Nov 26 2014 22:25
there isn't a substantial functional difference between a comm message that identifies itself as a reply in its content and a true reply
since a true reply is just a message that identifies itself as a reply by convention
Jonathan Frederic
@jdfreder
Nov 26 2014 22:28
@jasongrout by @minrk 's logic, I think what I do is fine, maybe it just needs to be less awkwardly implemented.
Min RK
@minrk
Nov 26 2014 22:29
"by @minrk's logic" - famous last words.
Jonathan Frederic
@jdfreder
Nov 26 2014 22:29
:P
Maybe I should be using guids instead of indicies to associate replies with requests...
Min RK
@minrk
Nov 26 2014 22:31
That's how we already do it with all existing req/rep messages
Jonathan Frederic
@jdfreder
Nov 26 2014 22:32
It's just implementing it as a lifo stack was less typing... And tcp is order guaranteed anyways
... which is what zmq and web sockets rely on, right?
Jason Grout
@jasongrout
Nov 26 2014 22:34
order is guaranteed in zmq and websockets
Jonathan Frederic
@jdfreder
Nov 26 2014 22:34
great
Jason Grout
@jasongrout
Nov 26 2014 22:37
yes, a comm message can say that it is a reply to a certain other comm message (by uuid, for example). But that info is already contained in the parent_msg header of the message underlying the comm message
or is @minrk saying that no messages from the browser have parent_msg attributes, basically?
Min RK
@minrk
Nov 26 2014 22:39
They absolutely do, the parent_msg should definitely be set if the message was a direct result of another message.
that's already used for handling all widget messages and output
so you probably don't need a new uuid
Jonathan Frederic
@jdfreder
Nov 26 2014 22:41
I can implement the resolution callbacks as dictionary of parent msg ids to callbacks.
I'll do that now, maybe it will make things less "janky"
Jonathan Frederic
@jdfreder
Nov 26 2014 22:47
@jasongrout I think I found a better way to implement it, I'll ping you when it's pushed
Jason Grout
@jasongrout
Nov 26 2014 23:28
@jdfreder: I wonder if it would be useful to put something in the routing machinery of comm messages so that you could send a message and right there register a callback to be called when a 'reply' comes, or a different callback if there is a timeout on waiting for a reply.
comm.send(msg, callback)
or hey, maybe comm.send could return a promise that is fulfilled/rejected when a reply comes. Or perhaps a comm.send_expect_reply(msg) returns such a promise.
Jonathan Frederic
@jdfreder
Nov 26 2014 23:36
@jasongrout I like the send_expect_reply idea
although if we do that we should implement something similar in Python
comms are so nice and symmetric now, I'd hate to break that
I pushed a new commit
overwriting the last
I think you'll like this one more
(than the previous)
Min RK
@minrk
Nov 26 2014 23:38
a comm message always results in at least two side-effect messages
(status: busy, status: idle)
same goes for all messages on the shell channel.
Kernel.send_shell_message takes callbacks and associates them with the parent ID
so that's already done
Comm.send passes through the same callbacks
Min RK
@minrk
Nov 26 2014 23:47
I guess there isn't a great way to share the same callback mapping in in the kernel and the comm manager
you could call Kernel.get_callbacks_for_msg(msg_id), but that may be entangling things too much
could be worth looking at, so that you don't need to reinvent the wheel