These are chat archives for DEAP/deap

1st
Jun 2016
Shivam Kalra
@shivamkalra
Jun 01 2016 17:48
Hi. Are there any examples on using Deap auto parallelism using MPI perhaphs using mpi4py library?
Yannick Hold
@soravux
Jun 01 2016 18:56
DTM ( http://deap.gel.ulaval.ca/doc/0.8/api/dtm.html ) was exactly what you described, but was abandoned. From its ashes was born SCOOP ( http://scoop.readthedocs.io/ ), which is based instead on ZeroMQ.
You may get lucky with some map()-interface wrapper around mpi4py such as [https://github.com/adrn/mpipool] or [https://github.com/twiecki/mpi4py_map], then just register the map() function to your toolbox and voilĂ  (e.g., http://deap.readthedocs.io/en/master/tutorials/basic/part4.html ).
François-Michel De Rainville
@fmder
Jun 01 2016 19:28
@wvangeit You almost got me to open an issue. If you go to the end of the script that displays the tree, you'll see that we re-evaluate every individual in the tree.
I admit that it is not ideal. I should add something to invalidate the fitness of the copied individual.
jrleek
@jrleek
Jun 01 2016 20:24
@shivamkalra I used MPI with DEAP, but not for doing the auto-parallelism. The auto-parallelism just speeds up evaluation, and I found that, in my case, using multiprocessing.map was fast enough that I could easily get selection to dominate the runtime. However, I did use MPI to set up multiple islands/demes. I can send you code if you're interested. It's not hard
Shivam Kalra
@shivamkalra
Jun 01 2016 20:48
@jrleek, Thanks for sharing information. I would like to see your code.
My fitness function is very expensive and involve manipulating image data. I would like to split fitness calculation on multiple nodes, may be your code can help me do that.