These are chat archives for ipython/ipython

22nd
Mar 2016
insomniacslk
@insomniacslk
Mar 22 2016 19:01
hello :) I have a question related to ipyparallel - I'd like to use the Client to parse a stream of tasks that is not entirely known beforehand. Looking at the docs and at the client's source code it seems to be non-trivial to do. Could someone give me advice on that?
Min RK
@minrk
Mar 22 2016 19:17
You can submit each task individually with apply
Or do it in chunks.
But you don't need to have a whole batch together before using IPython parallel.
insomniacslk
@insomniacslk
Mar 22 2016 19:46
thanks @minrk . Does apply still support the load balancing of the tasks?
I'd better explain my use case though
I have a framework that ingests processing tasks, starting from one single task. Then this task generates subtasks. I'd like to enqueue the newly generated tasks to the client and maintain the nice load balancing features
each task can have a very different footprint - it may be CPU-bound, I/O-bound, and may last from seconds to hours
so far I came up with this, after looking at the examples, https://gist.github.com/insomniacslk/ae390e04b8f06cd84286
I want to process them asynchronously and to enqueue them as soon as they are generated
insomniacslk
@insomniacslk
Mar 22 2016 19:51
replace the fibonacci function with whatever other function, e.g. time.sleep(random.randrange(10, 1000))
insomniacslk
@insomniacslk
Mar 22 2016 20:10
seems like it works :) thanks @minrk !