These are chat archives for thunder-project/thunder

8th
Mar 2015
Davis Bennett
@d-v-b
Mar 08 2015 00:44
@freeman-lab we need support in thunder for masking
been working on a hackish implementation of this with @Andrewosh, requires using rdd.join because filtering on keys is very slow
Jeremy Freeman
@freeman-lab
Mar 08 2015 00:46
@d-v-b can you clarify the use case?
Jeremy Freeman
@freeman-lab
Mar 08 2015 02:39
join will definitely be much slower than filtering, in general filtering should be quite fast
but depending on the desired use it might be easy to do it at another stage, e.g. during the images to series conversion
@tomsains you are right, the units for the fourier method are digital frequency in terms of "time points", so basically number of cycles per N time points, where N is the duration of the series
we are strongly considering adding an optional attribute of units to the TimeSeries class that would make it easy to describe outputs in more natural units of time
or maybe instead of units just a sampling rate
does something like that seem useful? anyone else have thoughts on this?
Jeremy Freeman
@freeman-lab
Mar 08 2015 02:45
also, thanks for the great summary of the EC2 installation! would welcome a PR with any additions to the official documentation that you think might clarify the process
Davis Bennett
@d-v-b
Mar 08 2015 03:15
@freeman-lab I wasn't getting good performance from something like data.filterOnKeys(lambda x: x in xyzs) where xyzs is a large set, but things greatly improved once we implemented xyzs as a broadcast variable, so ignore my earlier request for mask support :smiley_cat:
Jeremy Freeman
@freeman-lab
Mar 08 2015 04:31
Ah yes, for a large list without broadcasting you'll be serializing a copy out to all the tasks, definitely will be an improvement to bc it!
Jeremy Freeman
@freeman-lab
Mar 08 2015 09:57
@npyoung just commented on that issue with a couple ideas, curious to hear more about this speculation behavior, you added the speculation option to the standard thunder-ec2 launch configs? and what do you mean it was changing the partitioning?