These are chat archives for ipython/ipython

19th
Nov 2017
LordXyTh
@LordXyTh
Nov 19 2017 02:47
Hello everyone. Im trying to debug some code in ipython. It turns out that my lists contain data that shouldnt be there. Im wondering if there is a way I can set through the code and be able to see the list values to figure out whats going on within Ipython?
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:36
How is thread synchronization handled in ipyparallel client (ipyparallel.client.Client class)?
Client is handling all of the communication in a Tornado event loop running on a background thread. The user is simultaneously calling client's methods either directly or indirectly (using View, for example) from the main thread. For example the user can run Client.send_apply_request using DirectView.apply on the main thread while background thread is getting a reply to another apply request simultaneously. This can end up pretty badly. Am I missing something? What guarantees are there that we don't run into racing conditions or corrupted data in a typical use case like this? Am I missing something?
Min RK
@minrk
Nov 19 2017 21:42
The send actually occurs in the background thread.
The main thread builds the message, then hands it off to tornado to actually send.
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:44
Main thread also manipulates and reads the data structures of Client when doing that. The IOLoop thread may do this simultaneously.
Min RK
@minrk
Nov 19 2017 21:44
yup, and that's no problem.
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:44
how come?
Min RK
@minrk
Nov 19 2017 21:44
What's the problem you are envisioning?
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:45
for example:
Min RK
@minrk
Nov 19 2017 21:45
the background thread only populates results
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:46
let's see... code like this must be super carefully written ;)
i'll have a little look and get back with some questions
thanks for quick response!
okay found something
Client.send_apply_request
self.outstanding.add(msg_id)
this will be run from the main thread
Min RK
@minrk
Nov 19 2017 21:49
sure
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:49
Client._handle_apply_reply
self.outstanding.remove(msg_id)
will be run on the ioloop thread
Min RK
@minrk
Nov 19 2017 21:50
right
No problem
Remember, the Python interpreter is itself fundamentally threadsafe thanks to the GIL - no two threads can modify interpreter state concurrently via Python code.
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:55
so GIL locks python interpreter on each line of python code that gets executed?
if that is the case then i suppose this should work
seems a little bit dangerous though
Min RK
@minrk
Nov 19 2017 21:57

so GIL locks python interpreter on each line of python code that gets executed?

yes.

Sebastian Jylanki
@sjlnk
Nov 19 2017 21:57
thanks for your quick answers, much appreciated!
Min RK
@minrk
Nov 19 2017 21:57
Sure thing.
I'm done for tonight. If you find bugs you can trigger with real code, issues / pull requests are much appreciated.
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:58
i found something, not sure if it has been posted yet
Min RK
@minrk
Nov 19 2017 21:59
a test case would be the absolute best.
Thanks!
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:59
also wrote a resilient map scheduler on top of ipyparallel client
Min RK
@minrk
Nov 19 2017 21:59
cool!
Sebastian Jylanki
@sjlnk
Nov 19 2017 21:59
that works on dying enigines
dying/new engines
so you can run a long running map on amazon spot instances for example
saves a lot of money ;)
i can publish it if there is interest
client/launcher/scheduler combo connected with zmq
query state from scheduler using basic req/rep methods (rest-like)
get actual data from hub
using jupyter_client.session.Session for messaging