These are chat archives for thunder-project/thunder

23rd
Dec 2015
Ariel Rokem
@arokem
Dec 23 2015 00:00
@jmxpearson - I am intrigued, but could not find that repo on github. Maybe it's set to be private?
I have been working with a student doing some image processing for MRI with Spark (not Thunder though)
John Pearson
@jmxpearson
Dec 23 2015 01:22
@arokem Unfortunately, we don't have much worth showing yet. We have a machine image with FSL installed, but there's very little speedup over an SGE-style cluster, given that everything in FSL does disk i/o. We're currently working on just running some MVPA on Spark. That will be up and public as soon as it's functional. Sorry for any confusion.
Ariel Rokem
@arokem
Dec 23 2015 03:23
No problem. Looking forward to seeing it when it's ready to go!
Morgan Hough
@mhough
Dec 23 2015 06:18
thanks @d-v-b , @jmxpearson, @arokem. Yeah I would be very interested too. We would help with any existing effort if there seemed to be some fruitful directions to pursue. SGE and where appropriate GPU cover quite a bit of the bottlenecks. Forgive me for not knowing more yet but where does thunder really shine in terms of distributed operations? If we were doing very intensive times series modeling at every voxel would that leverage thunder to better advantage?
Jeremy Freeman
@freeman-lab
Dec 23 2015 20:35
@mhough @arokem @jmxpearson @d-v-b great discussion, thanks! a couple comments:
(1) there's basically nothing in thunder that's specific to a particular data type, and especially moving forward anything that is specific will live in separate modules
(2) it mainly shines when loading / processing / computing statistics on large collections of images and time series, especially if those operations can be expressed as local in space or local in time, and if you're doing some of each in a single analysis
@mhough your example of time-series modeling of every voxel definitely sounds kinda like that!
@arokem curious to hear what you've been up to with Spark + fMRI! is there a repo for any of it?
Ariel Rokem
@arokem
Dec 23 2015 21:47
No repo yet. The student who is working on this just submitted the paper as part of a course requirement. I hope that he will upload it some time in the new year. It's actually not an fMRI algorithm, but a denoising algorithm (this one: http://nipy.org/dipy/examples_built/denoise_nlmeans.html#example-denoise-nlmeans). Should be useful for other kinds of images as well. Will update when we have something...
But I am with you - don't see any reason not to use some variation on the regression tutorial on fMRI data