Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Kyle
@kr-hansen
It took 2 min and 10 sec using the multiprocessing library to load the 4000 individual tifs while it took 2 min 47 sec using the Thunder to load the 2 multipage tifs consisting of 2000 pages each.
However, it was much faster than the 6 min and 16 seconds to load the 4000 individual tifs using Thunder alone in local mode.
I imagine as it gets larger and I'm looking at my full dataset (26,600 timepoints saved over 13 multipage tifs) that difference might start to be noticeable when using multiprocessing to load individual tifs over the multipage tiff loading.
Davis Bennett
@d-v-b
if you're sticking to local mode I would use multiprocessing because thunder doesn't do any parallelization in local mode
but as soon as you're working with a cluster, thunder in spark mode will be faster
assuming you have 1 file per timepoint
Kyle
@kr-hansen
Ya, that's what I've gathered. The one issue I've run into using thunder in spark mode is that I can't do any frame-by-frame functions because the data are distributed.
Davis Bennett
@d-v-b
what do you mean by frame-by-frame functions?
Kyle
@kr-hansen
For example, if I want to do a homomorphic filter, a step in that would be to take my video, and subtract a spatially gaussian-filtered version of each frame from itself to remove the background.
I'll always get memory crashes because things are distributed and it tries to combine them on a single machine and the result is that it errors out.
for the subtraction.
Davis Bennett
@d-v-b
hm, that should be a simple map operation
Kyle
@kr-hansen
I know there are functions written into both thunder and bolt for this (.minus, etc...) but in practice they seem to crash with memory issues.
Davis Bennett
@d-v-b
first I would ignore as best you can the functions built in to thunder
like minus and plus and whatnot
Kyle
@kr-hansen
haha ok.
Davis Bennett
@d-v-b
the way I would do the filter you describe is write a function that takes an image an input and returns a filtered image as an output
Kyle
@kr-hansen
Well I've played around with using the standard map, which runs into issues.
Davis Bennett
@d-v-b
like lambda v: v - v.mean() will remove the mean from an image
Kyle
@kr-hansen
No, I'm able to create the filtered version just fine
Davis Bennett
@d-v-b
ah ok
so where is the problem?
Kyle
@kr-hansen
And I'm trying to do this prior to motion correction, so I'm interested in doing the frame by frame removal
So I can create a gaussian filtered array (4000, 1024, 1024) and I have my original array (4000, 1024, 1024)
But if I want to do original-gaussian, in a frame by frame manner, it doesn't work
I can do as you suggest, and take a single composite frame, to subtract from my original array, and that works ok.
Davis Bennett
@d-v-b
is your gaussian filter 3D or 2D?
Kyle
@kr-hansen
2D
2D spatial gaussian filter
Davis Bennett
@d-v-b
ok so you don't ever need a big gaussian filtered array
you need to gaussian filter each image independently
by using something like images.map(lambda v: v - gaussian_filter(v))
Kyle
@kr-hansen
Ok, I see what you're suggesting
Maybe I'll give that a try again.
I'm taking the processing pipeline we've been using in matlab in our lab and trying to convert it over to a Python mode, hence why I've been trying to use Thunder. The frame-by-frame portion was getting me stuck in Spark, so I shifted to try working only in local mode again, but this might still work in Spark mode.
Thanks. I'll try it out and report back for the benefit of others.
Davis Bennett
@d-v-b
no problem, and let me know if you have issues with image registration, that requires a few tricks
Kyle
@kr-hansen
@d-v-b I'll let you know if I have issues. When you talk about a few tricks, what do you mean? I was doing the registration about 6 months ago by taking a mean image as the reference to send to all the executors which seemed to work ok. Are you able to save the motion correction models as .jsons now? I was previously doing it with pickled files.
Davis Bennett
@d-v-b
the tricks apply if you want to do registration outside of the registration stuff built in to thunder
Kyle
@kr-hansen
so that's the second time you've mentioned using the functions outside of what is built in to thunder. Is thunder not quite being supported the same as it was 6 months-1 year ago? I've noticed the commits on the repo have basically stopped since a year or so ago.
Davis Bennett
@d-v-b
i think development has slowed down, but even if it was under active development I find it's much easier and flexible to use thunder for applying functions I wrote instead of using functions that are built in
in my case, I want to use a registration method that isn't built in to thunder so I don't have a choice
Kyle
@kr-hansen
@d-v-b So I went back to running it on Spark again and it seems to still be slightly faster to load the multi-page tiffs than the individual tiffs. I'm still loading the files from a standard file system since Thunder can't load tiffs from an hdfs. I imagine reading from the hdfs it might be quicker, but it seems like the multipage tiffs are still quicker.
Davis Bennett
@d-v-b
i don't understand how reading from a single file can be faster than reading from multiple files in a parallel context, but whatever works!
Kyle
@kr-hansen
It's a difference of 3-7 seconds or so from my testing, so not much, but it is faster in both cases on Spark than it is in local mode.
Gilles Vanwalleghem
@Yassum
Hello, so I was wondering if anyone has any idea how to produce these kind of plots : http://research.janelia.org/zebrafish/trajectories.html
As far as I see from the paper, that's supposed to be the online methods, but it's a bit sparse in details
mshahabi
@mshahabi
Hello all, I am new to thunder, I have a quick question I was wondering if thunder is an appropriate tool for applying large scale multi variable clustering with a mixture of spatial and temporal data... here is how my data looks like ['lable', 'Origin [lat, long]', 'Destination' [lat,long], 'distance', 'departure_time', 'total_travel_time', 'arrival_time',
]
mellertd
@mellertd
Hello everyone. My organization is trying to get Thunder going on a Yarn cluster using virtual environments. Has anyone tried this? It seems to be a little outside of the norm from what I have found, and it is certainly different from how things were done at Janelia when I was there. I am wondering whether this is an okay idea that we should continue to pursue
Boaz Mohar
@boazmohar
@mellertd As far as I understand, yarn mode is different for how you deploy your spark cluster, in Janelia it is in standalone mode. As far as thunder is concerned, they are both the same. You would need a Jupyter notebook and a way to get a spark context in it, which might be different then how it is done in Janelia. Is this the part you need help with?
mellertd
@mellertd
I guess the question had more to do with the virtual environment issue. I recall at Janelia, Thunder and dependencies were installed on all the nodes, so you could launch standalone clusters and everything worked. For various reasons, we don’t want to maintain installs on all the nodes and would rather let users manage their own environments. This is possibwith yarn mode, but it is rather clunky and is not officially supported in interactive mode (i.e. in Jupyter)