These are chat archives for ipython/ipython

7th
Mar 2017
Mesut Karakoç
@mkarakoc
Mar 07 2017 07:10
%time result = map(lambda x, y, z: x + y + z, range(1000), range(1000), range(1000))
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 163 µs
%time result = core.map_sync(lambda x, y, z: x + y + z, range(1000), range(1000), range(1000))
CPU times: user 8 ms, sys: 8 ms, total: 16 ms
Wall time: 14.3 ms
Hi! Why serial one is faster than parallel one in this case?
Davide Bortolami
@ddavidebor
Mar 07 2017 07:12
@mkarakoc It probably takes time to span the threads and allocate the memory
It's often not worth for very small and fast computations
Also
Mesut Karakoç
@mkarakoc
Mar 07 2017 07:15
If that is the case then, should I expect parallel one faster when I increase "range"?
Davide Bortolami
@ddavidebor
Mar 07 2017 07:16
I don't know how those functions are implemented in Python
But probably yes
If they use MKL acceleration it could behave differently
Mesut Karakoç
@mkarakoc
Mar 07 2017 07:17
I am now experimenting it and it seems still the same.
for "range(10000000)" serial: Wall time: 15.2 s and parallel: Wall time: 1min 15s
I know nothing yet about this. A real newbie... Thanks for answers.
Davide Bortolami
@ddavidebor
Mar 07 2017 07:22
@mkarakoc is lambda a very fast function to execute?
Mesut Karakoç
@mkarakoc
Mar 07 2017 08:03
May be I don't know or I don't understand what you mean.
Davide Bortolami
@ddavidebor
Mar 07 2017 13:15
The advantage of parallel processing is less evident with very small and quick functions
Mesut Karakoç
@mkarakoc
Mar 07 2017 13:16
I think I start to see the problem. Gnome System Monitor shows a full memory usage while CPU core's are not very busy. :)
Davide Bortolami
@ddavidebor
Mar 07 2017 13:16
And thus the overhead of spawning more threads becomes more relevant to the computation time
😀
Min RK
@minrk
Mar 07 2017 16:37
@mkarakoc to get good value out of parallelism, each assigned task needs to take more time than it takes to send the execution to the thread and retrieve the result.
So it's not just increasing the task count, it's increasing the task size.
One way to do this if your number of tasks is very large is to chunk the sequence, so each task is actually computing 1k or 1M elements in the list.