These are chat archives for thunder-project/thunder

23rd
Apr 2016
Jeremy Freeman
@freeman-lab
Apr 23 2016 21:37
@jwittenbach saw the local block stuff, awesome work!
one thought might be to rename LocalBlocks to LocalChunks
because from what I could tell it's less a local version of the existing Blocks class, but more a local version of an underlying ChunkedArray like structure that the generic Blocks can then wrap
as for the 'map_as_array' idea, not totally sure I'm seeing why we need that
Is it because otherwise there's no way to get back the result of an arbitrary per-block computation?
yeah I guess right now I'm solving that in thunder-extraction by dropping into the rdd am doing a map and a collect
Jeremy Freeman
@freeman-lab
Apr 23 2016 21:42
@stevevarner @jwittenbach cool discussion I'm all for adding a topickle and frompickle method on both images and series that dumps the entire object to a pickled file, and uses local or spark methods where appropriate
so in the spark case it's basically be exactly what @stevevarner suggests and would be an easy way to provide HDFS persistence
If someone wants to open an issue I can take a stab at the implementatipn