These are chat archives for evhub/coconut

8th
Jul 2016
Constantine Molchanov
@moigagoo
Jul 08 2016 06:51
@evhub Just tested it, and it looks great so far! Hitting 150 URLs with plain map$(requests.get) takes a minute, parallel_map$(requests.get) takes 15 seconds (4x less since there're 4 cores), concurrent_map$(requests.get) takes 6-7 seconds.
Evan Hubinger
@evhub
Jul 08 2016 08:49
@moigagoo Great! I've always liked the idea of using map as a functional parallelization primitive. Good to see it's working well, and it'll definitely make its way into the next release (along with recursive_iterator and probably some improved lambda syntax), although that might be a little bit since I just released v1.1.1.
Constantine Molchanov
@moigagoo
Jul 08 2016 09:10

@evhub I wonder how an event loop-based map implementation could land in Coconut. Maybe, async_map? Sounds clear. However, in this context,concurrent_map would look weird IMHO. Maybe asyncio_map, with an emphasis on that it's best suitable for IO-bound tasks? But in this case, there will be a module name in the decorator name, which may be misinterpreted. Another option is to replace threads with event loop right in concurrent_map, but that would make it unusable with Python 2.

I think it's more of a political decision, one you as a BDFL should make :-)

Evan Hubinger
@evhub
Jul 08 2016 17:11
@moigagoo I'm definitely open to it, but I'm not sure what the use cases would be. What benefits would an asyncio-based map provide, do you think?
Constantine Molchanov
@moigagoo
Jul 08 2016 18:57
@evhub Asynchronous approach is useful when handling IO-bound tasks, i.e. lots of small IO transactions each of which is cheap CPU-wise. Typical examples are sending HTTP requests, writing to and reading from a DB. The point is that you can have relatively small number of threads—a few thousand tops on a decent Mac—whereas you can have much more open sockets (each socket corresponding to an IO transaction). So a thread-based web server could handle a few thousand concurrent requests but an event loop-based one could handle hundreds of thousands of them.