These are chat archives for thunder-project/thunder

26th
Mar 2017
Davis Bennett
@d-v-b
Mar 26 2017 01:26
@tom-f-oconnell I don't know of anyone doing something like this, and it's probably not going to come from thunder devs any time soon
Jason Wittenbach
@jwittenbach
Mar 26 2017 08:03
@tom-f-oconnell we use NumPy for the local back-end on Thunder, and we wrote Bolt to serve as the Spark-based back-end. I think the use case that you mention would be a great addition and it would effective involve writing an out-of-core backend based on something like Dask (http://dask.pydata.org/en/latest/)
Jason Wittenbach
@jwittenbach
Mar 26 2017 08:11
Since Bolt copies the NumPy API to a large extent, the code in Thunder is either agnostic to which backend is being used, or it can be handled with simple switches. So if a backend with an API similar to Bolt were built, but that relied on something like Dask (which handles lazy out-of-core operations), then Thunder could be easily extended to use it as a backend as well
Davis Bennett
@d-v-b
Mar 26 2017 15:18
this sounds totally awesome
was there discussion before about hooking dask / lazy array operations into thunder? I know the guys at elastik have some kind of lazy numpy array ops library: https://github.com/ilastik/lazyflow
Tom O'Connell
@tom-f-oconnell
Mar 26 2017 20:49
yea, i was looking through the issues and saw mention of dask before for a similar use case
in issue #342
i hadnt seen lazyflow before, so i'll give that a look. https://soft-matter.github.io/pims/v0.3.3/ claims to support lazy loading images in to numpy arrays, but it seemed to not behave this way in practice for the one case i tested (my use case)