hello :) I have a question related to ipyparallel - I'd like to use the Client to parse a stream of tasks that is not entirely known beforehand. Looking at the docs and at the client's source code it seems to be non-trivial to do. Could someone give me advice on that?
thanks @minrk . Does apply still support the load balancing of the tasks?
I'd better explain my use case though
I have a framework that ingests processing tasks, starting from one single task. Then this task generates subtasks. I'd like to enqueue the newly generated tasks to the client and maintain the nice load balancing features
each task can have a very different footprint - it may be CPU-bound, I/O-bound, and may last from seconds to hours