These are chat archives for freeman-lab/zebra

9th
Dec 2015
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:14
@d-v-b, @jwittenbach , what's a good way to obtain a time-lapse MIP of fish brain data? Have you done this? This must be simple, but I am not as good in Thunder as I should be..
Jason Wittenbach
@jwittenbach
Dec 09 2015 16:15
MIP?
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:15
max-intensity projection
over z
Jason Wittenbach
@jwittenbach
Dec 09 2015 16:16
so you just want an array that’s (t, x, y) ?
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:17
so, taking raw stack each time point, flattening it over Z using Max intensity projection, and then saving the resulting time series of 2D images to a file
yes, exactly
Jason Wittenbach
@jwittenbach
Dec 09 2015 16:17
ah, I see
you can get a good amount of the way to that with:
imgs = tsc.loadImages(…)
max_proj = imgs.maxProjection(axis=2)
for saving the results, you could maybe do it in parallel
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:19
imgs is time series, or raw RDD?
Jason Wittenbach
@jwittenbach
Dec 09 2015 16:19
imgs and max_proj will both be Thunder Images objects
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:19
OK, sound good, will try. Thanks!
Jason Wittenbach
@jwittenbach
Dec 09 2015 16:20
then you would write a function that takes each record and writes out the image
something like
def write_image(record):
   k, v = record[0], record[1]
   plt.imshow(v)
   plt.savefig(“path_to_location/image”+k+”.eps)
and then apply this funtion:
max_proj.apply(write_image).count()
the count will just force it to actualy evaluate everything
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:23
can I convert max_proj to numpy array using pack()?
like MIPS=max_proj.toTimeSeries().applyValues(lambda x: x).pack()?
Nikita Vladimirov
@nvladimus
Dec 09 2015 16:55
yes, this works. Thanks, @jwittenbach !
Jason Wittenbach
@jwittenbach
Dec 09 2015 17:28
Yep, that should work! No need to go all the way TimeSeries even, you can just collect max_proj and that should be faster
Davis Bennett
@d-v-b
Dec 09 2015 18:20
@nvladimus I have a notebook that does this and saves as a movie, saving as tifs would be just as simple
Nikita Vladimirov
@nvladimus
Dec 09 2015 18:34
@jwittenbach TimeSeries.pack() seems more convenient because it returns numpy array, while max_proj.collect() returns a list, if I get it correctly.
Jason Wittenbach
@jwittenbach
Dec 09 2015 18:34
oh, right
Nikita Vladimirov
@nvladimus
Dec 09 2015 18:35
@d-v-b I am saving to TIFF, thanks.
Jason Wittenbach
@jwittenbach
Dec 09 2015 18:35
try max_proj.values().collect()
Nikita Vladimirov
@nvladimus
Dec 09 2015 18:35
ok
aha, another magic word which I should know, .values() :)
Jason Wittenbach
@jwittenbach
Dec 09 2015 18:36
of course, if you want to compute something like dF/FdF/F, then you’ll have to go to Series anyway
Davis Bennett
@d-v-b
Dec 09 2015 18:36
... I have a notebook that does all this
do you want it?
Jason Wittenbach
@jwittenbach
Dec 09 2015 18:37
though I’m currently working on a way to not need to go all the way to Series if you just want to do something like ΔF/F\Delta F/F and then come back to Images
should make creating movies a little faster
Nikita Vladimirov
@nvladimus
Dec 09 2015 18:40
@d-v-b sure, please share. Thanks!
some of this won't be useful for you
Davis Bennett
@d-v-b
Dec 09 2015 18:57
yeah apologies this code is not well documented, but the core functionality of taking dff (in an Images only way, to avoid converting series to images conversion) is there
and it saves the max projections in binary format, so that you don't have to do the cluster-heavy calculations more than once
the second half of the notebook is ugly stuff for making videos with matplotlib
but that can be easily replaced with tif writing via skimage.external.tifffile
Nikita Vladimirov
@nvladimus
Dec 09 2015 18:59
yes, I just save results in TIFF and use FiJi then. Thanks.
Davis Bennett
@d-v-b
Dec 09 2015 19:05
nikita, what compression ratio do you usually get with bzip2?
Davis Bennett
@d-v-b
Dec 09 2015 19:11
i'm asking because it might be possible to do image compression using the tif format
since you can specify a compression level when you save as tif
the compressed tif stacks in my testing are ~40% the size of the original
but they remain fully accessible to image readers, unlike zipped images
Nikita Vladimirov
@nvladimus
Dec 09 2015 19:30
about 30%
is it loss-less compression?
Davis Bennett
@d-v-b
Dec 09 2015 19:32
i believe so
Nikita Vladimirov
@nvladimus
Dec 09 2015 19:32
actually, more like 35% for bz2
Davis Bennett
@d-v-b
Dec 09 2015 19:32
the tif compression tag allows lossy and lossless compression
Nikita Vladimirov
@nvladimus
Dec 09 2015 19:33
if it's lossless and 40%, this would be really good for us. But can Thunder load compressed TIFF files?
Davis Bennett
@d-v-b
Dec 09 2015 19:35
i need to test this too, but anything that loads tif files should be able to load compressed tif files
....i will test it if I can get space on the cluster :(
Davis Bennett
@d-v-b
Dec 09 2015 20:35
compressed tifs can be read by spark