These are chat archives for thunder-project/thunder
.tiffile at a time, so I don't really what will happen if you try to parallelize reading data from a small number of files
multiprocessinglibrary to parallelize over cores
multiprocessinglibrary doesn't work to parallelize over reading in
.tiffiles. After starting a parallel pool, and feeding in
p.map(td.images.fromtif, [rawimagesdir])it takes just as long to load as if I'd run
td.images.fromtif(rawimagesdir), but also errors out. If anyone is aware of a way to parallelize local loading of individual
tiffiles, I'd be interested to know of it.
skio.imreaddoesn't take a directory as input, so I'll have to iterate over the files to some degree
4000+.tifs where each one is a single timepoint
skio.imreadand iterating over each individual
.tifin the new folder is faster in a local method, and only slightly faster than Thunder reading in the multipage
multiprocessinglibrary to load the 4000 individual
tifswhile it took 2 min 47 sec using the Thunder to load the 2 multipage tifs consisting of 2000 pages each.
tifsusing Thunder alone in local mode.
multiprocessingto load individual
tifsover the multipage tiff loading.
thunderdoesn't do any parallelization in local mode
sparkmode will be faster
sparkmode is that I can't do any frame-by-frame functions because the data are distributed.
boltfor this (
.minus, etc...) but in practice they seem to crash with memory issues.
lambda v: v - v.mean()will remove the mean from an image
images.map(lambda v: v - gaussian_filter(v))