These are chat archives for ipython/ipython

14th
Apr 2015
Jason Grout
@jasongrout
Apr 14 2015 13:43 UTC
@minrk - thanks!
Bas Nijholt
@basnijholt
Apr 14 2015 14:15 UTC

I try to start and run engines on the nodes of a hpc which works with PBS. I followed the documentation on creating a profile with a file PBSEngineSetLauncher and PBSControllerLauncher. I'm starting by first ssh’ing to the server and then run $ipython notebook --profile=pbs --port=9876 --no-browser. Then I create a tunnel to the server with: $ssh -NL 9876:localhost:9876 hpc05 and and then in the browser cluster tab (http://localhost:9777/tree#clusters) start the PBS profile.

in a IPython notebook I then type parallel.Client()

and get the following error

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-12-cc08b7633683> in <module>()
----> 1 parallel.Client()

/home/mwimmer/python/env_stable/lib/python2.7/site-packages/IPython/parallel/client/client.pyc in __init__(self, url_file, profile, profile_dir, ipython_dir, context, debug, sshserver, sshkey, password, paramiko, timeout, cluster_id, **extra_args)
    409             )
    410 
--> 411         with open(url_file) as f:
    412             cfg = json.load(f)
    413 

IOError: [Errno 2] No such file or directory: u'/home/bnijholt/.ipython/profile_pbs/security/ipcontroller-client.json
Jason Grout
@jasongrout
Apr 14 2015 15:16 UTC
@minrk - also IPython.utils.path.get_ipython_dir -> IPython.paths.get_ipython_dir
Damian Avila
@damianavila
Apr 14 2015 17:33 UTC
/@all, can't connect now... have a good meeting!!
Jason Grout
@jasongrout
Apr 14 2015 18:49 UTC
@jdfreder - are you available to have a discussion about a detail of widget syncing?
Jonathan Frederic
@jdfreder
Apr 14 2015 18:54 UTC
I'm on the phone with Brian right now
in a few minutes, yes
Jason Grout
@jasongrout
Apr 14 2015 19:01 UTC
never mind. Turns out the interesting idea I had wasn't better than what we are doing after all.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02 UTC
@jasongrout okay, sorry : (
I'm available to chat now
Jason Grout
@jasongrout
Apr 14 2015 19:02 UTC
no problem.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02 UTC
if you change your mind
Jason Grout
@jasongrout
Apr 14 2015 19:02 UTC
the issue was the property lock on the python side
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02 UTC
Ah
Jason Grout
@jasongrout
Apr 14 2015 19:02 UTC
and the default behavior to call str() on any object that is not jsonable to get a json representation
so basically, what happens is that a message comes in from js, and the property lock gets set
Jonathan Frederic
@jdfreder
Apr 14 2015 19:03 UTC
yes
Jason Grout
@jasongrout
Apr 14 2015 19:03 UTC
then it's from_json-ed, and possibly does lots of things, including possibly triggering a set of the same property
so if the name matches the property lock, we to_json the value and check it against the property lock
Jonathan Frederic
@jdfreder
Apr 14 2015 19:04 UTC
but if it's not serializable by json, the value is always Object+pointer or hash (I forget which)
which doesn't change, if it's pointer
erm-
always changes
Jason Grout
@jasongrout
Apr 14 2015 19:05 UTC
we are clearly talking about two different things
if it is not serializable, then str() is called on it
(or maybe repr())
nothing about object+pointer or hash
so if I try to send a random object out in a widget message, I'll get a str/repr of the object going across the wire
Anyway, so 2 problems:
  1. if what came in was a list, then likely the trait property is that list, which is mutable, so it can change and not be sent back out because the lock is changed (the lock isn't a copy, it's a reference to the json value that came in)
  1. if our to_json(value) gives back an object, it will be compared with the string representation that came in from js
so the comparison basically is doing my_object == str(my_object), which of course fails
(usually)
so, question 1: why do we default to calling str/repr of un-jsonable objects?
Jason Grout
@jasongrout
Apr 14 2015 19:11 UTC
I remember thinking about that before. Now I think it is probably a bad idea to not throw an error and make the user give something that is truly jsonable
Jonathan Frederic
@jdfreder
Apr 14 2015 19:14 UTC
mmm, I'm not sure. What's wrong with changing https://github.com/jupyter/jupyter_notebook/blob/master/jupyter_notebook/widgets/widget.py#L288 so the from_json call is made before the lock?
Jason Grout
@jasongrout
Apr 14 2015 19:14 UTC
ah, yes, that was my idea from before.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:14 UTC
i.e.
                json_value = sync_data[name]
                from_json = self.trait_metadata(name, 'from_json', self._trait_from_json)
                value = from_json(json_value)
                with self._lock_property(name, value):
                    setattr(self, name, value)
Jason Grout
@jasongrout
Apr 14 2015 19:15 UTC
so basically you want to compare the python values, not the json values?
The problem there is that almost always, mutable values will pass the comparison
Jonathan Frederic
@jdfreder
Apr 14 2015 19:16 UTC
Alternatively, when the lock is checked go from Python value -> json
Jason Grout
@jasongrout
Apr 14 2015 19:16 UTC
because the from_json returns a mutable python value. Things possibly change on that value, and then the lock compares the value with the lock
Jonathan Frederic
@jdfreder
Apr 14 2015 19:16 UTC
and store the json value in the lock
Jason Grout
@jasongrout
Apr 14 2015 19:16 UTC
store the json string, or the json value?
Jonathan Frederic
@jdfreder
Apr 14 2015 19:17 UTC
Store json_value
Jason Grout
@jasongrout
Apr 14 2015 19:17 UTC
(json value is the python list/dict)
right, we do store the json value now.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:18 UTC
Ah, yes
Jason Grout
@jasongrout
Apr 14 2015 19:18 UTC
yes
and we can't store the string since dict order could get messed up
Jonathan Frederic
@jdfreder
Apr 14 2015 19:18 UTC
ah
ok
I see
Jason Grout
@jasongrout
Apr 14 2015 19:19 UTC
We could do json.loads(json.dumps(to_json(value)))in the check
Jonathan Frederic
@jdfreder
Apr 14 2015 19:20 UTC
You mean json.dumps(json.loads(to_json(value))) ?
Jason Grout
@jasongrout
Apr 14 2015 19:20 UTC
no, what I said
actually, that wouldn't work either.
because the default in IPython is to call str/repr on objects it can't encode
they would throw errors in json.dumps
Jonathan Frederic
@jdfreder
Apr 14 2015 19:23 UTC
I'm meeting someone for lunch,
I'll be back in 40 min.
I'm going to think about it some more
So -- what's the rationale behind calling repr() on arbitrary objects to send them over the wire? Is that useful? In this case for us, it encouraged sloppy programming to not give back real json from the to_json method.
I would much rather that the user has to be explicit about how they are encoding nonjsonable objects.
Jason Grout
@jasongrout
Apr 14 2015 19:31 UTC
oh, I guess we could dump both the lock and the value to json strings using the sort_keys=True option of the python json encoder.
(but we still have the problem of json_clean automatically calling repr on objects)
Thomas Kluyver
@takluyver
Apr 14 2015 19:56 UTC
@jdfreder You mentioned in the hackpad moving IPython.paths back to utils? Why do you want to do that? I moved it out of utils because I don't think it belongs there.
Jonathan Frederic
@jdfreder
Apr 14 2015 20:11 UTC
@ellisonbg suggested it, I was just taking notes : )
@jasongrout I'm back
@takluyver but IIRC he thinks they are utility functions, just ones that should be highly visible.
Thomas Kluyver
@takluyver
Apr 14 2015 20:14 UTC
utils is mostly utilities in the sense of things that are not specific to IPython
Jonathan Frederic
@jdfreder
Apr 14 2015 20:15 UTC
Ah
Thomas Kluyver
@takluyver
Apr 14 2015 20:15 UTC
It always seemed strange to me that things like get_ipython_dir() were in utils
and almost none of the other code in there is specific to IPython
sysinfo gets a couple of IPython specific things
I'm sure you can find other examples, but not very many. That's the principle I think we should use for what goes in utils.
Min RK
@minrk
Apr 14 2015 20:31 UTC
@takluyver that makes more sense than the private API point I was making this morning.
Jason Grout
@jasongrout
Apr 14 2015 21:41 UTC
@jdfreder - I've got to go, but I'm interested in what else you want to say about the widget syncing tomorrow