These are chat archives for ipython/ipython

7th
Aug 2017
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:03
hello all!
I'm trying to test that an ipython magic works inside of jupyter by doing this: jupyter nbconvert --execute --to html tests/tmp.ipynb, where the tmp.ipynb imports %foobar, which is not a real magic. Unexpectedly, even though the output html says ERROR:root:Line magic function%foobarnot found., the jupyter nbconvert command itself does not crash, even though it's supposed to if any exceptions arise. If it doesn't crash when it's supposed to, I can't reliably test whether a magic is in fact working. Any idea how to make it crash properly?
Min RK
@minrk
Aug 07 2017 14:33
What does the magic do?
You might have better luck testing with IPython directly, rather than invoking all of the extra Jupyter machinery, which shouldn't be necessary, depending on your magic.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:33
@minrk the magic i'm ultimately trying to test is something we built inhouse at spotify
%bigquery, talks to GCP bigquery
Min RK
@minrk
Aug 07 2017 14:34
Jupyter has no awareness of magics, so it's probably easiest to isolate the test in an in-process IPython API call.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:34
won't ipython itself have the same behavior?
of not-crashing when it can't successfully import a magic?
or is that just a jupyter thing
Min RK
@minrk
Aug 07 2017 14:34
it may, but it's easier to catch
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:35
good point
Min RK
@minrk
Aug 07 2017 14:35
The error handling in magics is probably an IPython thing
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:35
i went thru jupyter because i wasn't sure how to test ipython magic directly
Min RK
@minrk
Aug 07 2017 14:35
For a long time, the only thing that mattered for IPython was what it looked like.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:35
any good starting points for how to "test in an in-process IPython API call"?
yeah that makes sense =)
Min RK
@minrk
Aug 07 2017 14:36
But then Jupyter came around, and it now makes a difference whether an error was raised, or just 'reported' to output. This is the kind of thing that affects nbconvert's "this failed"
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:36
ah right
Min RK
@minrk
Aug 07 2017 14:36
But there are still a number of 'errors' that are caught and displayed as nicer errors than tracebacks, but now Jupyter frontends don't know that the cell failed.
You can also try something like nbval which runs through notebooks and can do things like output comparison.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:37
gotcha, everything is illuminated =)
interesting
i'm just looking for the simplest thing that will throw a unix syscall >0
the python API approach seems good
Min RK
@minrk
Aug 07 2017 14:39
We are working on removing these as we find them, so if you open an Issue on ipython/ipython about 'magic not found' results in reporting execute success, it should hopefully get fixed.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:39
okay happy to do so
not familiar with the IPython API.. is there a "run cell" equivalent function call in there?
Min RK
@minrk
Aug 07 2017 14:41
Once an InteractiveShell is instantiated: shell.run_cell('cell contents')
e.g. shell = IPython.core.InteractiveShell.instance()
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:46
thanks!
ipython/ipython#10736
^^ issue'd ^^
looks like the API changed IPython.core.interactiveshell.??
Min RK
@minrk
Aug 07 2017 14:49
oh, right. Missed a module level.
Juan-Pablo Velez
@jpvelez
Aug 07 2017 14:49
ah got it
thanks!
Juan-Pablo Velez
@jpvelez
Aug 07 2017 15:04
In [5]: shell.run_cell('%bigquery') ERROR:root:Line magic function `%bigquery` not found. Out[5]: <ExecutionResult object at 7f279a590790, execution_count=None error_before_exec=None error_in_exec=None result=None>
the ERROR:root:Line magic function%bigquerynot found. is just printed
is there any way to capture this so i can evaluate it?
Min RK
@minrk
Aug 07 2017 15:05
What test runner are you using? Most have useful output-capturing utilities
Juan-Pablo Velez
@jpvelez
Aug 07 2017 15:06
bats
mostly testing cli functionality
good point i'll try that
Min RK
@minrk
Aug 07 2017 15:06
I don't know bats!
Sylvain Corlay
@SylvainCorlay
Aug 07 2017 15:59
@minrk we are making some progress on the xeus-cling kernel
Completion and help:
Peek 2017-07-27 15-56.gif
Widgets:
widgets.gif
I am trying to get my head around the binary buffers part of the protocol in ipykernel. A lot of serialize.py seems to be about using binary serialization for long sequences for ipyparallel.
Jason Grout
@jasongrout
Aug 07 2017 16:05

I am trying to get my head around the binary buffers part of the protocol in ipykernel

The binary buffers in messages? Like the wire format?

Sylvain Corlay
@SylvainCorlay
Aug 07 2017 16:07
not really
in this case, some buffers are computed
and appended to the message
        buffers = serialize_object(data,
            buffer_threshold=session.buffer_threshold,
            item_threshold=session.item_threshold,
        )
        content = json_clean(dict(keys=list(data.keys())))
then a send_message with content=content, buffers=buffers is sent.
so, does serialize_object remove keys from data ?
Sylvain Corlay
@SylvainCorlay
Aug 07 2017 16:14
nevermind, reading up more code.
Sylvain Corlay
@SylvainCorlay
Aug 07 2017 16:37
Related note: Jason just pointed to me the PR of Vidar on another channel.
Jason Grout
@jasongrout
Aug 07 2017 16:39
It's several levels above what you are dealing with - likely not helpful to understand things
Juan-Pablo Velez
@jpvelez
Aug 07 2017 19:12
hello!
anybody know how to starts an InteractiveShell with the iPython API that uses an ipython_config in a profile?

import IPython
from IPython.core.profiledir import ProfileDir
from IPython.paths import locate_profile

def test_analytics_toolbox_bigquery_magic(capsys):

profile_dir = ProfileDir().find_profile_dir(locate_profile())
shell = IPython.core.interactiveshell.InteractiveShell.instance(profile_dir=profile_dir)
magic_name = 'bigquery'
magic = shell.find_magic(magic_name)
assert magic is not None

i thought it would be by supplying the profile dir, but this isn't working
Juan-Pablo Velez
@jpvelez
Aug 07 2017 19:31
import IPython
from IPython.core.profiledir import ProfileDir
from IPython.paths import locate_profile

def test_analytics_toolbox_bigquery_magic(capsys):

  config = IPython.core.application.ProfileAwareConfigLoader('ipython_config.py', locate_profile()).load_config()
  shell = IPython.core.interactiveshell.InteractiveShell.instance(config=config)
  magic_name = 'bigquery'
  magic = shell.find_magic(magic_name)
  assert magic is not None
The config is successfully loaded, but it's still not finding the magic
when i paste this code into an ipython session, it works
when i call this from pytest, it doesn't, even though the config is correctly loaded
Sylvain Corlay
@SylvainCorlay
Aug 07 2017 20:48
Quick question on the signing of messages in ipykernel:
does the signature include the binary buffers?