These are chat archives for thunder-project/thunder
thunder-extractionusing the same NMF code as the paper from Pnevmatikakis et al. (https://github.com/epnev/ca_source_extraction)? It says they provided it to both SIMA and Thunder. However, looking through the code had me guessing that was pre Thunder 1.0. Is their CNMF the NMF that got transferred over, or is it just a generic NMF for testing purposes?
thunder-extractionat one point, so I was wondering if that is the version that made it in.
imagespersonally, but I don't see any reason programmatically why Thunder would be the bottleneck/limitation for a series. I work with images that are 1024x1024 pixels recorded at 20 fps for extended periods of time (40,000+ frames in some cases) and Thunder works for what I need. I'd suggest you try it out.
spark.yarn.executor.memoryOverhead=1024. However, I'm still getting some types of memory issues related to the shuffle calls going from images to series on my cluster. I don't have a solution to that issue yet but am working on it
image.toseries()you might improve performance by changing the
sizeargument to something much smaller like
boltand then get rid of the implementations currently in
element_wisefunction for the mapping. That was cleaner to follow than what I did for
bolt, so probably some combination of the two would be best overall
images.valuesbasically the same as the underlying
images.valuesis either a
ndarraydepending on whether you're in local or distributed mode
pandasif you've used that
pandasmuch, but good to know. I'm coming from a Matlab world...
thunderto work in Spark?
DaskArrayis a lot like the
engineyou'd have an
images.valuesthat's either backed by spark (via bolt), by a local numpy array, or by a dask array
boltfor Spark use at some point though probably, right? I just don't want to embark on adding more of the
ndarraylike functions to
boltif it could eventually be piped in via something like Dask that would be added.
boltin the short term couldn't hurt