These are chat archives for ipython/ipython

14th
Apr 2015
Jason Grout
@jasongrout
Apr 14 2015 13:43
@minrk - thanks!
Bas Nijholt
@basnijholt
Apr 14 2015 14:15

I try to start and run engines on the nodes of a hpc which works with PBS. I followed the documentation on creating a profile with a file PBSEngineSetLauncher and PBSControllerLauncher. I'm starting by first ssh’ing to the server and then run $ipython notebook --profile=pbs --port=9876 --no-browser. Then I create a tunnel to the server with: $ssh -NL 9876:localhost:9876 hpc05 and and then in the browser cluster tab (http://localhost:9777/tree#clusters) start the PBS profile.

in a IPython notebook I then type parallel.Client()

and get the following error

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-12-cc08b7633683> in <module>()
----> 1 parallel.Client()

/home/mwimmer/python/env_stable/lib/python2.7/site-packages/IPython/parallel/client/client.pyc in __init__(self, url_file, profile, profile_dir, ipython_dir, context, debug, sshserver, sshkey, password, paramiko, timeout, cluster_id, **extra_args)
    409             )
    410 
--> 411         with open(url_file) as f:
    412             cfg = json.load(f)
    413 

IOError: [Errno 2] No such file or directory: u'/home/bnijholt/.ipython/profile_pbs/security/ipcontroller-client.json
Jason Grout
@jasongrout
Apr 14 2015 15:16
@minrk - also IPython.utils.path.get_ipython_dir -> IPython.paths.get_ipython_dir
Damian Avila
@damianavila
Apr 14 2015 17:33
/@all, can't connect now... have a good meeting!!
Jason Grout
@jasongrout
Apr 14 2015 18:49
@jdfreder - are you available to have a discussion about a detail of widget syncing?
Jonathan Frederic
@jdfreder
Apr 14 2015 18:54
I'm on the phone with Brian right now
in a few minutes, yes
Jason Grout
@jasongrout
Apr 14 2015 19:01
never mind. Turns out the interesting idea I had wasn't better than what we are doing after all.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02
@jasongrout okay, sorry : (
I'm available to chat now
Jason Grout
@jasongrout
Apr 14 2015 19:02
no problem.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02
if you change your mind
Jason Grout
@jasongrout
Apr 14 2015 19:02
the issue was the property lock on the python side
Jonathan Frederic
@jdfreder
Apr 14 2015 19:02
Ah
Jason Grout
@jasongrout
Apr 14 2015 19:02
and the default behavior to call str() on any object that is not jsonable to get a json representation
so basically, what happens is that a message comes in from js, and the property lock gets set
Jonathan Frederic
@jdfreder
Apr 14 2015 19:03
yes
Jason Grout
@jasongrout
Apr 14 2015 19:03
then it's from_json-ed, and possibly does lots of things, including possibly triggering a set of the same property
so if the name matches the property lock, we to_json the value and check it against the property lock
Jonathan Frederic
@jdfreder
Apr 14 2015 19:04
but if it's not serializable by json, the value is always Object+pointer or hash (I forget which)
which doesn't change, if it's pointer
erm-
always changes
Jason Grout
@jasongrout
Apr 14 2015 19:05
we are clearly talking about two different things
if it is not serializable, then str() is called on it
(or maybe repr())
nothing about object+pointer or hash
so if I try to send a random object out in a widget message, I'll get a str/repr of the object going across the wire
Anyway, so 2 problems:
  1. if what came in was a list, then likely the trait property is that list, which is mutable, so it can change and not be sent back out because the lock is changed (the lock isn't a copy, it's a reference to the json value that came in)
  1. if our to_json(value) gives back an object, it will be compared with the string representation that came in from js
so the comparison basically is doing my_object == str(my_object), which of course fails
(usually)
so, question 1: why do we default to calling str/repr of un-jsonable objects?
Jason Grout
@jasongrout
Apr 14 2015 19:11
I remember thinking about that before. Now I think it is probably a bad idea to not throw an error and make the user give something that is truly jsonable
Jonathan Frederic
@jdfreder
Apr 14 2015 19:14
mmm, I'm not sure. What's wrong with changing https://github.com/jupyter/jupyter_notebook/blob/master/jupyter_notebook/widgets/widget.py#L288 so the from_json call is made before the lock?
Jason Grout
@jasongrout
Apr 14 2015 19:14
ah, yes, that was my idea from before.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:14
i.e.
                json_value = sync_data[name]
                from_json = self.trait_metadata(name, 'from_json', self._trait_from_json)
                value = from_json(json_value)
                with self._lock_property(name, value):
                    setattr(self, name, value)
Jason Grout
@jasongrout
Apr 14 2015 19:15
so basically you want to compare the python values, not the json values?
The problem there is that almost always, mutable values will pass the comparison
Jonathan Frederic
@jdfreder
Apr 14 2015 19:16
Alternatively, when the lock is checked go from Python value -> json
Jason Grout
@jasongrout
Apr 14 2015 19:16
because the from_json returns a mutable python value. Things possibly change on that value, and then the lock compares the value with the lock
Jonathan Frederic
@jdfreder
Apr 14 2015 19:16
and store the json value in the lock
Jason Grout
@jasongrout
Apr 14 2015 19:16
store the json string, or the json value?
Jonathan Frederic
@jdfreder
Apr 14 2015 19:17
Store json_value
Jason Grout
@jasongrout
Apr 14 2015 19:17
(json value is the python list/dict)
right, we do store the json value now.
Jonathan Frederic
@jdfreder
Apr 14 2015 19:18
Ah, yes
Jason Grout
@jasongrout
Apr 14 2015 19:18
yes
and we can't store the string since dict order could get messed up
Jonathan Frederic
@jdfreder
Apr 14 2015 19:18
ah
ok
I see
Jason Grout
@jasongrout
Apr 14 2015 19:19
We could do json.loads(json.dumps(to_json(value)))in the check
Jonathan Frederic
@jdfreder
Apr 14 2015 19:20
You mean json.dumps(json.loads(to_json(value))) ?
Jason Grout
@jasongrout
Apr 14 2015 19:20
no, what I said
actually, that wouldn't work either.
because the default in IPython is to call str/repr on objects it can't encode
they would throw errors in json.dumps
Jonathan Frederic
@jdfreder
Apr 14 2015 19:23
I'm meeting someone for lunch,
I'll be back in 40 min.
I'm going to think about it some more
So -- what's the rationale behind calling repr() on arbitrary objects to send them over the wire? Is that useful? In this case for us, it encouraged sloppy programming to not give back real json from the to_json method.
I would much rather that the user has to be explicit about how they are encoding nonjsonable objects.
Jason Grout
@jasongrout
Apr 14 2015 19:31
oh, I guess we could dump both the lock and the value to json strings using the sort_keys=True option of the python json encoder.
(but we still have the problem of json_clean automatically calling repr on objects)
Thomas Kluyver
@takluyver
Apr 14 2015 19:56
@jdfreder You mentioned in the hackpad moving IPython.paths back to utils? Why do you want to do that? I moved it out of utils because I don't think it belongs there.
Jonathan Frederic
@jdfreder
Apr 14 2015 20:11
@ellisonbg suggested it, I was just taking notes : )
@jasongrout I'm back
@takluyver but IIRC he thinks they are utility functions, just ones that should be highly visible.
Thomas Kluyver
@takluyver
Apr 14 2015 20:14
utils is mostly utilities in the sense of things that are not specific to IPython
Jonathan Frederic
@jdfreder
Apr 14 2015 20:15
Ah
Thomas Kluyver
@takluyver
Apr 14 2015 20:15
It always seemed strange to me that things like get_ipython_dir() were in utils
and almost none of the other code in there is specific to IPython
sysinfo gets a couple of IPython specific things
I'm sure you can find other examples, but not very many. That's the principle I think we should use for what goes in utils.
Min RK
@minrk
Apr 14 2015 20:31
@takluyver that makes more sense than the private API point I was making this morning.
Jason Grout
@jasongrout
Apr 14 2015 21:41
@jdfreder - I've got to go, but I'm interested in what else you want to say about the widget syncing tomorrow