These are chat archives for thunder-project/thunder

19th
Apr 2016
Chris Tech
@techchrj
Apr 19 2016 16:17
I agree with @kkcthans that the refactoring and structure of 1.0.0 makes much more sense and is much easier to follow. As for my question, has anyone noticed a massive slowdown in performance when comparing version 1.0.0 to version 0.6?? I recently integrated v1.0.0 into our spark application that was previously using v0.6 and the performance is on the order of 10x slower. A process that took 6 seconds before is now taking 58secs to complete. We are using the image series and spark i.e. td.images.fromtif(values, engine=sc). Our spark cluster has not changed. Are their performance properties that we should be tweaking to speed this up to what it was? Any help would be greatly appreciated.
Jeremy Freeman
@freeman-lab
Apr 19 2016 16:25
@techchrj thanks! I'd love to get to the bottom of any performance regression, there's no reason anything should be slower even with just default settings
we are doing validation queries in a few places on things like data shape to improve error messaging but that may have inadvertently slowed down a couple workflows
but it should be easy to get your performance back to what it was as nothing fundamental has changed
can you post a gist or the exact workflow? If it's just loading tifs we should be able to replicate with our own data
Chris Tech
@techchrj
Apr 19 2016 16:39

@freeman-lab Sure thing. Here is one of the flows, we are using LZW compressed 32bit floating point TIFs (each is 900x900 pixels and we are looking at anywhere from 100 to >5000 images) :

Old way:
from PIL import Image
....tsc is thunder context c
imagedata = tsc.loadImages(images, inputFormat='tif')
minimage = imagedata.min()
Image.fromarray(minimage)

New Way (which is much slower):

imagedata = td.images.fromtif(images, engine = sc)
minimage = imagedata.min()
Image.fromarray(minimage)

As you can see, it is very straight forward and not much in for a code change but performance is way different.

Jeremy Freeman
@freeman-lab
Apr 19 2016 17:06
@techchrj cool this is great, we'll start by confirming the difference in our setup, what are the rough image dimensions
Chris Tech
@techchrj
Apr 19 2016 17:07
@freeman-lab images are 900x900 pixels. 32bit floating point TIFs
Jeremy Freeman
@freeman-lab
Apr 19 2016 17:26
@techchrj awesome that'll be easy to replicate
@jwittenbach want to take a stab at this? we can use some of @sofroniewn's new data :)
initial thoughts are that there's possibly an extra 'first' call to validate the image dimensions, but weird that'd account for such a large difference
and maybe something silly is happening at the end, cause before .min() resulted in a local array whereas here the result of .min() is still images and we do a final .toarray()
oh the other thing to check is the parallelism, forget if and when the default changed
Chris Tech
@techchrj
Apr 19 2016 17:34
@freeman-lab "the other thing to check is the parallelism, forget if and when the default changed" -- do you mean the "spark.default.parallelism" property. We have that set in our spark configuration and I actually played with that property today to see if it had any effect on Thunder performance, and I did not see anything that substantial.
Jeremy Freeman
@freeman-lab
Apr 19 2016 17:35
sorry was talking about the npartitions keyword argument for images.fromtif()
The default is either the number of images or the default parallelism of the cluster
and it may have changed since 0.6
we can play with this in our test setup to see if that's it
Chris Tech
@techchrj
Apr 19 2016 17:37
that would be great.!! thanks. I'll do some digging as well.
@freeman-lab Appreciate anything you guys find out!
Jeremy Freeman
@freeman-lab
Apr 19 2016 17:43
totally, really good to know about this! we'll figure it out :)
Jason Wittenbach
@jwittenbach
Apr 19 2016 17:56
@freeman-lab @techchrj happy to look into this — I’ll keep you posted on what I find
Nicholas Sofroniew
@sofroniewn
Apr 19 2016 18:17
@jwittenbach also happy to help - let me know if you need anything
i'm recording 1792*1682 uint16 tiffs right now