These are chat archives for thunder-project/thunder

19th
Mar 2015
Ben Poole
@poolio
Mar 19 2015 00:38
anyone come across the use case of wanting to sample a handful of frames form a dataset? current loading methods only support start/stopIdx and not subsample. I also don't see a way to randomly take a sample without instantiating the RDD - both sample() and filter() load all the data first.
Jeremy Freeman
@freeman-lab
Mar 19 2015 02:17
Interesting, this is for loading images?
One option would be to add support for loading arbitrary lists of image indices (relative to the order in the file listing)
Does that seem useful?
wolfbill
@wolfbill
Mar 19 2015 03:06
Hi guys
which version of thunder includes tsc.loadBinary
Ben Poole
@poolio
Mar 19 2015 03:27
@freeman-lab yes, loading a subset of images to compute either some simple statistics (mean, variance) or to fit a more complex model that doesn't take into account time (RASL low-rank basis). It's also useful for creating a quick video to see the activity across an experiment without loading a potentially massive dataset. I think the usecase of supporting arbitrary list of image indices and supporting a subsampling factor argument will both come up. If you want to do subsampling and only support indices then you need to get a count of the images first (which will naively involve loading the full RDD with loadImages())
wolfbill
@wolfbill
Mar 19 2015 03:30
I'd like to get a old version of thunder ? Anyone can help me ?
wolfbill
@wolfbill
Mar 19 2015 03:33
@poolio thank you
wolfbill
@wolfbill
Mar 19 2015 06:35
Hi I'd like to run the examples:data=tsc.loadBinary('data'),I don't know where the data is
see the section about sample data
wolfbill
@wolfbill
Mar 19 2015 06:55
@d-v-b Thank you
Jeremy Freeman
@freeman-lab
Mar 19 2015 14:37
@poolio great points, our image loading strategy (both for NFS and S3) should give us the full list of to-be-loaded images without actually loading them
which means that we should be able to do both indexing and subsampling ahead of time without any loading
Ben Poole
@poolio
Mar 19 2015 18:01
@freeman-lab yeah I was just looking at that. Thinking of just adding in more kwargs and changing selectByStartAndStopIndices to selectIndices. Will submit a PR