These are chat archives for thunder-project/thunder

12th
Jul 2016
Jason Wittenbach
@jwittenbach
Jul 12 2016 01:20
@Brocoli_GT_twitter With the new version of Thunder (1.0), you should it install it however you would install any other Python package. I’m not an expert on how that works on the Databricks platform. Perhaps you could zip the repository and then the platform has some way to upload it?
@joe311 Both versions are run through the same set of tests, so they should both be good to go! If you don’t have any other constraints, I would choose 3.0, as eventually everything will go that way :smile:
@d-v-b what kinds of errors are you getting?
Davis Bennett
@d-v-b
Jul 12 2016 14:49
@jwittenbach the key error might be this: File "/groups/ahrens/home/bennettd/bolt/bolt/spark/chunk.py", line 259, in <lambda> partitioner = lambda k: ravel_multi_index(k[s], ranges) ValueError: invalid entry in coordinates array
Jeremy Freeman
@freeman-lab
Jul 12 2016 14:50
@d-v-b @jwittenbach is localcorr even supposed to support 3D data? i didn't remember that
could be the local version supports it by accident
maybe @sofroniewn remembers, i think he wrote the local one
Davis Bennett
@d-v-b
Jul 12 2016 14:53
iirc localcorr never had a problem with volumetric data. The only dimension-dependent part should be the blurring, right?
Jeremy Freeman
@freeman-lab
Jul 12 2016 14:55
ah yes you're right
hm not clear then, if it's in chunk i suspect @jwittenbach will know what's up
are you using the latest thunder and bolt?
Davis Bennett
@d-v-b
Jul 12 2016 14:55
ya
speaking of localcorr, could we directly calculate the correlation without converting to series?
since the correlation is basically a normalized dot product, and the dot product should be doable as a reduce over images, right?
this would avoid a shuffle, I think
Jason Wittenbach
@jwittenbach
Jul 12 2016 15:17
@d-v-b are you doing any masking or anything? also, the rest of the stack trace might be helpful. Looks like it’s failing while trying to do the new shuffling operation with the custom partitioner. It’s possible that localcorr is somehow using the keys in a way that I didn’t anticipate...
Jeremy Freeman
@freeman-lab
Jul 12 2016 15:18
@jwittenbach i'd check that concatenate still works as expected, issue is probably there
requires that the input into fromrdd has correct key structure
Davis Bennett
@d-v-b
Jul 12 2016 15:33
@jwittenbach all i'm running is the code I pasted in there
Jeremy Freeman
@freeman-lab
Jul 12 2016 15:39
this td.images.fromrandom(shape=[10,10,10,10], engine=sc).localcorr() yeah?
Davis Bennett
@d-v-b
Jul 12 2016 15:42
yep
after importing thunder as td obvs