These are chat archives for ipython/ipython
c.TaskScheduler.hwm=1. This is the default. This ensures that only one task can be assigned to each node at a time, so no engine should be idle unless all of your tasks are being worked on.
Our setup is as follows:
4 remote machines running 128 engines and 1 machine running the controller
We are using the default profile without any modification
Credentials files are shared through share drive.
We are able to process our requests completely without any issue.
However, once all the requests are processed and engines remain idle for about 1 hr,
we are not able to process our requests without restarting the ip controller and engines.
In jupyter notebook it hungs at the cell where import of libraries are done.