These are chat archives for thunder-project/thunder

12th
May 2016
mobcdi
@mobcdi
May 12 2016 10:44
Hi all, Came across your project as I'm about to deploy a spark cluster for a video mining project (both for the first time). I hope you don't mind me asking a bunch of questions.
*Does Thunder work well with OpenCV?
  • Do cluster managers like mesos help or hinder Thunder?
    *Any advice for dealing with videos over 1 hour in duration and that come in different resolutions?
    I was hoping I could use Thunder and Spark to preprocess the videos but not sure where to start
Davis Bennett
@d-v-b
May 12 2016 15:32
@mobcdi what kind of preprocessing do you need to do? Regarding Thunder working with opencv, I'm not up to date on the python wrappings for opencv but I think any opencv operation you can apply to a single frame of video should be easy to parallelize with thunder.
mobcdi
@mobcdi
May 12 2016 16:27
Hi thanks @d-v-b for responding. Honestly I'm not sure what preprocessing is needed as it's my first time working with videos or images for that matter.
For example Is it possible to resize an entire video using thunder so I have all my videos with a common width x height and is that important first step before comparing other dimensions
Davis Bennett
@d-v-b
May 12 2016 16:53
@mobcdi if the frames of each video can be stored as numpy arrays, than yes resizing them with thunder should be very simple
mobcdi
@mobcdi
May 12 2016 16:58
Seems possible. Any recommendations for resources for processing videos as sources for data mining? Have plenty for numeric datasets
Davis Bennett
@d-v-b
May 12 2016 17:06
I think it totally depends on what processing you want to do, but for very basic array manipulations (e.g., resizing) and image operations like filtering and whatnot numpy and sci-kit image are both good