These are chat archives for ipython/ipython

9th
Apr 2015
Antonino Ingargiola
@tritemio
Apr 09 2015 00:07
Thinking of monkey patching, seems that .preprocess_cell() does already the right thing.
This message was deleted
So is probably .run_cell() that does not raise the exception.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 00:15
Yeah, I think specifically you want to catch the message that includes the error
The outputs get appended to a list here, so that is probably where you need to check if it is an error or not
Antonino Ingargiola
@tritemio
Apr 09 2015 00:23
I should wait for the notebook execution to finish and go and read the first error.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 00:54
@tritemio You don’t need to wait for the entire notebook to finish executing
The way it works is that the execute preprocessor sends a request to the kernel to execute the cell, and then it waits for replies from the kernel
Some of those replies are the outputs of the cell
So if one of the outputs is of type error, then you know the cell has thrown an error
At which point you can stop executing and raise an error in the execute preprocessor
Antonino Ingargiola
@tritemio
Apr 09 2015 01:11
@jhamrick , thanks I'm trying exactly what you suggest
Antonino Ingargiola
@tritemio
Apr 09 2015 01:18
done, it works!
Jessica B. Hamrick
@jhamrick
Apr 09 2015 01:18
Awesome!
Antonino Ingargiola
@tritemio
Apr 09 2015 01:22
thanks @jhamrick , here it is the gist
I realized that there is a simpler solution, I was patching both run_cell() and preprocess_cell() but the latter is enough
Antonino Ingargiola
@tritemio
Apr 09 2015 01:28
def preprocess_cell(self, cell, resources, cell_index):
    """
    Apply a transformation on each code cell. See base.py for details.
    """
    if cell.cell_type != 'code':
        return cell, resources

    outputs = self.run_cell(cell)
    cell.outputs = outputs

    for out in outputs:
        if out.output_type == 'error':
            self.log.error('Error executing the following cell:')
            self.log.error(str(cell.source))
            raise RuntimeError('Error executing the notebook.')

    return cell, resources
Jessica B. Hamrick
@jhamrick
Apr 09 2015 01:33
Yeah, that looks great
I don’t know if you’d necessarily want to log the whole cell source. It might be better to put it in the runtime error
Antonino Ingargiola
@tritemio
Apr 09 2015 01:35
Yeah, the ideal would be re-raise the same exception, but I don't know if it's possible.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 01:35
I don’t think we’d want to actually re-raise the same exception
Since technically the exception didn’t occur in the same process
But the traceback probably should be reported somewhere for sure
Antonino Ingargiola
@tritemio
Apr 09 2015 01:37
I see, you are right, would be nice to propagate the traceback so it can be displayed in the calling notebook
I was doing this with runipy
how can i have a file saved on the server side instead of ‘rendered’ in the output cell ?
instead of printing ‘file_widget.value’ i should add a save method, right ?
Sylvain Corlay
@SylvainCorlay
Apr 09 2015 03:56
This message was deleted
(sorry, wrong chat)
Scott Sanderson
@ssanderson
Apr 09 2015 04:52
is there any difference between ipython_config.py and ipython_kernel_config.py?
Kyle Kelley
@rgbkrk
Apr 09 2015 04:57
Is there any way to suppress logs in the notebook from another library's logging?
Min RK
@minrk
Apr 09 2015 05:37
@ssanderson ipython_config affects terminal IPython, kernel_config only when it's a kernel
e.g. notebook
Scott Sanderson
@ssanderson
Apr 09 2015 12:39
@rgbrk if it's using python's default logging you can set a handler on the library's logger to pipe it somewhere other than stdout
If you come up with a better fix let me know...I have the same problem. I updated requests and its log lines are crashing users' browsers...I wonder how hard it'd be to implement throttling for kernel messages
Anthony Louis Burns, Esq.
@anthonylouisburns
Apr 09 2015 14:08
ok this is still very rough - but I am pretty proud of it - its a proof of concept for displaying ipython notebooks on github pages http://anthonylouisburns.github.io/jupyter-pages/
https://github.com/anthonylouisburns/jupyter-pages
any thoughts much appreciated - I am going to add functionality for importing notebooks
Scott Sanderson
@ssanderson
Apr 09 2015 14:56
so, I figured out why my kernels start slowly...I'm setting a configurable object that has a moderately large (10 MB) memory footprint, and it's being deepcopyed 58 times in the course of starting a kernel
seems like it might be beneficial to make it possible to flag a configurable as "do not copy"
Scott Sanderson
@ssanderson
Apr 09 2015 15:03
also, unrelated, but if there are IPython folks around at PyCon this week I'd love to meet up at some point
Brian E. Granger
@ellisonbg
Apr 09 2015 16:01
Kyle and Thomas are there at pycon
Scott Sanderson
@ssanderson
Apr 09 2015 16:36
@ellisonbg @takluyver @rgbkrk cool! myself and @llllllllll will be hanging out all week; are you guys doing a sprints session?
(yes, Joe's github name is 10 lower case L's)
we also just re-wrote our own version of the first-time experience tour for IPython, some part of which would probably be useful for the main project; the big new thing is that it actually walks the user through the process of executing cells and explains the notion of cell outputs with simple examples
would love to get your guys' thoughts on it
Min RK
@minrk
Apr 09 2015 16:39
@ssanderson sounds great.
@ssanderson I have an idea for the config copying, but need to finish the big split first.
Scott Sanderson
@ssanderson
Apr 09 2015 16:41
@minrk cool! I'm working around it for the time being by implementing a deepcopy that just returns self
__deepcopy__*
Min RK
@minrk
Apr 09 2015 16:42
@ssanderson what version are you using? When fixing your deepcopied logger problem, I think that meant that only copy is called (not deepcopy) on the final objects.
Scott Sanderson
@ssanderson
Apr 09 2015 16:43
@minrk I'm on ~2 months before 3.0 with a bunch of specific PRs pulled in; in the process of actually upgrading to 3.1, so that may fix the issue too
Min RK
@minrk
Apr 09 2015 16:43
ah, ok. Let me know if it doesn't. Changing the copy is probably something we should do, anyway.
Scott Sanderson
@ssanderson
Apr 09 2015 16:44
I think it makes sense to copy for mutable attributes of classes that are going to be instantiated repeatedly
but lots of configurables are used as singletons
Min RK
@minrk
Apr 09 2015 16:46
It's really only primitive containers (list, dict, etc.) that we care about copying.
But I expect our config copying pattern is not ideal in a few ways, and can be improved.
Scott Sanderson
@ssanderson
Apr 09 2015 16:47
would it be feasible to add a config_copy=True|False flag to TraitType or somesuch?
Min RK
@minrk
Apr 09 2015 16:47
And in those cases, deep copy is specifically not what we want.
I don't think we need that.
Scott Sanderson
@ssanderson
Apr 09 2015 16:47
or do you think there's a more substantial revision to be made
Min RK
@minrk
Apr 09 2015 16:48
never objects, types, etc.
If anything, I think it should be simpler than a new option - be more judicious about what we copy.
Scott Sanderson
@ssanderson
Apr 09 2015 16:48
my gut is that copying by default is reasonable, since it's usually the safe thing to do
and the failure mode when not copying is incorrect will be really confusing
because you'll mutate some instance attribute of your configurable object and it'll have unexpected side-effects on another instance
you're probably right that copy will often be preferable to deepcopy though
Min RK
@minrk
Apr 09 2015 16:51
And that's what it does now, I think - deepcopy of a config object deep copies the config structure, and stops with a shallow copy of each final trait.
Min RK
@minrk
Apr 09 2015 17:03
@/all should IPython Kernel examples follow the ipython_kernel repo, or stay with IPython'? I'm somewhat inclined to leave them with IPython, since they really cover how to use IPython in general.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:04
You thinking about the nb examples?
Min RK
@minrk
Apr 09 2015 17:04
yup
everything in examples/IPython Kernel
Brian E. Granger
@ellisonbg
Apr 09 2015 17:05
IIRC the kernel examples are really ipython kernel specific things
Min RK
@minrk
Apr 09 2015 17:05
It also covers magics, !, IPython execution
Brian E. Granger
@ellisonbg
Apr 09 2015 17:05
Eventually, I think we want the "notebook" docs to be truly language independent and/or multi-language
Our docs are quite a mess now - but I think most of the stuff in the kernel subdir should go with ipython
Min RK
@minrk
Apr 09 2015 17:07
It's partly a question of how we view the IPython kernel repo - is it a prominent repo that end-users should be aware of?
Or should general 'using IPython' docs go together, however you are using it?
Brian E. Granger
@ellisonbg
Apr 09 2015 17:07
I think so - I would expect all kernels to have docs that describe how to use that kernel
Eventually the main notebook docs should just link to the individual kernel docs
But I think IPython should be like other kernels in that respect
Min RK
@minrk
Apr 09 2015 17:08
ok, so leave them all with ipython, take no usage examples with the kernel
Brian E. Granger
@ellisonbg
Apr 09 2015 17:08
The only difference is that I think some of our notebook docs will have python examples, but we should strive to mix that up
But it is probably a good time to rethink our organization of docs based on the disucssions we had 2 dev meetings ago.
Min RK
@minrk
Apr 09 2015 17:08
@ellisonbg there's an important difference between IPython and other kernels that's relevant here - the IPython kernel is two repos
Brian E. Granger
@ellisonbg
Apr 09 2015 17:09
I don't think it is a good idea for each new repo to have both sphinx and nb based docs
Mmm, can you clarify that? Not quite following...
How is the ipython kernel 2 repos?
Min RK
@minrk
Apr 09 2015 17:10
  1. IPython itself, where all the interesting stuff is
  2. ipython_kernel, a separate repo where the protocol implementation is
Matthias Bussonnier
@Carreau
Apr 09 2015 17:10
IPython repo, or it will break most of the already existing links.
for the time beeing
Brian E. Granger
@ellisonbg
Apr 09 2015 17:10
Ahh, I forgot that the kernel was separate from the command line main ipython
Matthias Bussonnier
@Carreau
Apr 09 2015 17:10
we can slowly move them with time
Brian E. Granger
@ellisonbg
Apr 09 2015 17:11
Hmm, that makes it a bit weird then
For example, where is IPython.display going?
And all of the magics that we ship?
Min RK
@minrk
Apr 09 2015 17:11
it's staying in core, because it's used by plain terminal IPython
Magics, too, are already organized
The code split is already done, I'm just asking about examples.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:12
So ipython-kenel will depend on ipython?
Min RK
@minrk
Apr 09 2015 17:12
yes, and jupyter-client, which holds the basic implementation of the message spec (session).
Brian E. Granger
@ellisonbg
Apr 09 2015 17:12
I am just trying to catch up on the code split part :)
Got it.
Then I think we can put user focused docs for ipython and ipython-kernel in the ipython repo
Min RK
@minrk
Apr 09 2015 17:13
Great, that's what I was thinking.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:13
Because all of the APIs those docs will describe will be in ipython - plus that is our existing repo right?
Min RK
@minrk
Apr 09 2015 17:13
yes
ipython_kernel is just pulling out the kernel-side parts of IPython.kernel
Brian E. Granger
@ellisonbg
Apr 09 2015 17:14
But I think we should get rid of the name "examples" - they aren't that, they are nb based docs
Got it
Another question - should we think about shipping docs inside the packages so they get installed?
Min RK
@minrk
Apr 09 2015 17:14
Yup, and I think we discussed moving a lot of the general stuff to a docs-only jupyter-doc repo.
but that wouldn't include IPython usage docs.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:15
We should probably talk about the docs side of things in a dev meeting soon
Min RK
@minrk
Apr 09 2015 17:15
History of examples isn't the most important thing, so we can move files around after the split, I'm just trying to get a starting point, so I can run the splits. Each split takes about half an hour to run, and the kernel is next.
Sure
Maybe even a supplementary Thursday meeting for just that, so it's not competing for time with other updates.
Should I be merging my own PRs for the split stuff?
Brian E. Granger
@ellisonbg
Apr 09 2015 17:16
Even if we cant' completely reinvent and rewrite out docs it would be (pausing that though - yes I agree that the history of docs is not too important) good to at least think about how we want our docss are organized
I am fine with you merging those
We can iterate as needed
Min RK
@minrk
Apr 09 2015 17:17
At this point, it's pretty serialized - each PR depends on the last.
Sure
Brian E. Granger
@ellisonbg
Apr 09 2015 17:17
It is 10am Th right now - are folks around for a short meeting? @/all ?
Min RK
@minrk
Apr 09 2015 17:18
I think F's in Colorado, Thomas and Kyle are at PyCon.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:18
Ahh, thats right
Matthias Bussonnier
@Carreau
Apr 09 2015 17:18
There just me also.
Min RK
@minrk
Apr 09 2015 17:18
Probably not enough people for that conversation.
Maybe next week.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:18
Yep, I agree
Matthias Bussonnier
@Carreau
Apr 09 2015 17:18
so you can take control of Saul and hang around.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:19
If I didn't have meetings all day :)
Hiring is a meeting-generating activity
Matthias Bussonnier
@Carreau
Apr 09 2015 17:19
Well, you take control of Saul, and say "can't join, am in Berkeley right now"
Brian E. Granger
@ellisonbg
Apr 09 2015 17:20
If I can't eat the cupcakes though, it doesn't count as my being there
Matthias Bussonnier
@Carreau
Apr 09 2015 17:20
Fair enough.
Min RK
@minrk
Apr 09 2015 17:22
holy crap, submodules. Git is so bad at them.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 17:24
Are submodules going to go away after the split?
Min RK
@minrk
Apr 09 2015 17:27
yes
Running the notebook from master will require fetching dependencies with bower.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:28
Yeah!!!
Jessica B. Hamrick
@jhamrick
Apr 09 2015 17:28
Woohoo! Yeah, submodules are the worst.
Brian E. Granger
@ellisonbg
Apr 09 2015 17:28
I won't miss them a bit
Min RK
@minrk
Apr 09 2015 17:28
Which will bring new headaches all its own, of course, but we've been pretending to not have a js toolchain a little too long.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 17:28
They seem like such a great idea at first, but then you try to use them.
Yeah, I think it’s reasonable to require the dependencies be installed with bower.
Min RK
@minrk
Apr 09 2015 17:29
It makes perfect sense, it's just git's super bad at it. So may operations seem to simply ignore them, and leave them in the wrong state.
Jessica B. Hamrick
@jhamrick
Apr 09 2015 17:29
Yeah
Min RK
@minrk
Apr 09 2015 17:30
I just did a rebase with no commits that touch the submodule, but because the branch I was rebasing onto had moved the submodule, every single commit had a conflict on the submodule that was fixable with git reset HEAD path/to/submodule
Jonathan Frederic
@jdfreder
Apr 09 2015 17:35
Wow, it looks like by the end of the day the split will be done
Matthias Bussonnier
@Carreau
Apr 09 2015 17:35
I use gulp a bit with phosphor. and I must say that livereload is nice in developping.
Jonathan Frederic
@jdfreder
Apr 09 2015 17:35
I use gulp with poster too, it's nice
Matthias Bussonnier
@Carreau
Apr 09 2015 17:35
everytime you save, your browser refresh, or even just get the new js version injected.
feel closer to interactive developpment.
Min RK
@minrk
Apr 09 2015 17:35
End of the day might be ambitious, but probably tomorrow.
There will still be a lot of work to do, but the repos will be separated, at least.
Matthias Bussonnier
@Carreau
Apr 09 2015 17:36
@jdfreder Did you acchived to get livereload working without browser extension ?
Jonathan Frederic
@jdfreder
Apr 09 2015 17:38
No I haven't
Did you?
Matthias Bussonnier
@Carreau
Apr 09 2015 17:38
No. I have to click the ext to enable it.
apparently you can use express as a middleware that inject a script tag to only reload part of the JS.
But extension is fine, only one click away everyday.
Jonathan Frederic
@jdfreder
Apr 09 2015 17:39
but I usually don't use the watch feature of gulp
I found that I save frequently
sometimes I save with a typo
and it causes watch to crash
so I have to go to the kernel terminal to restart it
Matthias Bussonnier
@Carreau
Apr 09 2015 17:42
You can set-up watch not to crash.
IIUC.
I mean at least with typescript.
Min RK
@minrk
Apr 09 2015 17:42
We didn't mention environment variables for JUPYTER_DATA_DIR and JUPYTER_RUNTIME_DIR. Do we want those, in addition to JUPYTER_CONFIG_DIR?
Nicholas Bollweg
@bollwyvl
Apr 09 2015 17:46
@epifanio I needed widget uploading, too, along with more file metadata... i hacked that example to use FileReader.readAsDataURL: http://nbviewer.ipython.org/gist/bollwyvl/5b2d233143c54982acb8
I've written a really ghetto live-coding thing that just uses the bare upstreams, no gulp: i find it to be quite sufficient for many small tasks:
the core of it is just a tiny bit of bash:
```bash
#!/bin/bash -e
for i in $(seq 1 $#); do
  npm run ${!i} &
  PIDS[$i]=$!
done

echo $PIDS

trap "kill ${PIDS[*]}" SIGINT

wait
of course, it doesn't do any complex chaining... but then many of the tools have so much complexity in them already (watchers, post-processors, plugins) that you don't really need much
Jonathan Frederic
@jdfreder
Apr 09 2015 17:59
@bollwyvl cool!
Scott Sanderson
@ssanderson
Apr 09 2015 18:37
@minrk I usually rebase out any submodule changes I have before rebasing onto a branch that's moved a submodule
and then add a commit that just updates the submodule to what I want it to be
Min RK
@minrk
Apr 09 2015 18:37
@ssanderson there were no submodule changes in the rebased commits
Not a single file or top-level directory in common.
Scott Sanderson
@ssanderson
Apr 09 2015 18:38
hrm, that's surprising
we use a TON of submodules right now because we're deploying custom versions of lots of stuff
Min RK
@minrk
Apr 09 2015 18:38
I agree!
Scott Sanderson
@ssanderson
Apr 09 2015 18:38
but I haven't hit that issue you described
Min RK
@minrk
Apr 09 2015 18:39
When they don't move, it's generally not a problem. It just seems to handle moving really badly (as in, not at all).
Scott Sanderson
@ssanderson
Apr 09 2015 18:39
if I get a conflict it's usually because I commited a submodule on the branch and then rebased
Sylvain Corlay
@SylvainCorlay
Apr 09 2015 18:41
@minrk is it a good time to repoen PR s on jupyter notebook?
Min RK
@minrk
Apr 09 2015 18:42
@SylvainCorlay no, wait for it to be in its new repo.
Sylvain Corlay
@SylvainCorlay
Apr 09 2015 18:43
ok thanks
Min RK
@minrk
Apr 09 2015 21:13
@ellisonbg I'm getting ready to split nbconvert. There are two notebooks and one rst file of nbconvert docs. Do we want these in the nbconvert repo, or in the common jupyter-docs destination?
Min RK
@minrk
Apr 09 2015 23:19
@ellisonbg same question for notebook docs and the jupyter_notebook repo, actually. I need to know what examples, docs should go with jupyter_notebook before I split. That's really the last thing in the way of finishing the split.
Guilherme Freitas
@gpfreitas
Apr 09 2015 23:33
I was thinking how nice it would be to have a variable name "x" turn red as I type "x = " in the IPython notebook if I have already defined x elsewhere, or if it's defined in a cell above the current one (even if the cell is currently unexecuted). I would love to contribute but I have never contributed to an open source project, so I might need some hand holding. Any pointers to how this may be achieved?
Matthias Bussonnier
@Carreau
Apr 09 2015 23:39
if cell is unexecuted then you need to get your hand dirty with JS.
In your particular case, I wonder if this is really desirablem as you can often use temp variable.
anyway..
the highlighting in general is done by codemirror : http://codemirror.net/
codemirror define "modes" the parse/lex a string and return css classes.
what you want to do is improve python mode to detect already existing variables.
Guilherme Freitas
@gpfreitas
Apr 09 2015 23:50
I imagined that I'd need JS for the unexecuted cells issue. Is there any specific part of the IPython codebase/docs that would be a good starting point? Or particularly instructive tutorial on JS + IPython Notebook (I have seen many blog posts about hacking the notebook via JS, but maybe there is a particularly good one)?
Guilherme Freitas
@gpfreitas
Apr 09 2015 23:56
Also, if you know which one will be easier to tackle first (highlight pre-existing variable, or highlight variable present in cell above), please let me know!