These are chat archives for thunder-project/thunder
So I've tried running the setup.py from the source distribution of the latest version of Thunder (0.5.1) while I had Spark 1.5.0 and Anaconda 2.2.0 active. Just as before, it seems I can open Thunder correctly with spark and anaconda active in my files. Is there anywhere in the setup scripts that I need to define what versions of Spark/Hadoop/Python to look for before running setup.py build and setup.py install?
It seemed to setup correctly with the files being placed in the appropriate folder. However I can't load the fish-series example. Working with the IT guys on the cluster, they suggested it is a Hadoop dependency issue, but trying to build from source with Spark 1.5.0 and Anaconda 2.2.0 from the hadoop dependencies didn't seem to solve the problem.
Any suggestions on how to get Thunder running on a cluster with Hadoop 2.6 installed?
Imagesobject, the key represents the frame, which is just an integer
sortByKey()is one function where is actually relies on the keys being tuples
data = tsc.loadImages(…) result = data.applyKeys(lambda k: (k,)).filterOnKeys(…).sortByKey()
sortByKey()gives the error whether or not I call
filterOnKeys()first, so hopefully this will solve your problem from now until we can fix this
scaleparameter in the calls to
Colorizeto see if you can’t get some brigher colors — the
polarcolorization scheme maps the combined magnitudes of the weights to the “value” (i.e. bright to dark), so your small weights are all just coming out as variations on black.
amax) for the final plot. If the individual images in your dataset do not have a simliar relationship, then this might not be appropriate for you.
tsc.loadExampleassumes that the example files are in a fixed location within the Thunder directory, but the relative path to this folder might be different on different machines in your Spark network (driver, master, worker, etc)
path_to_thunder/utils/data/mouse/images/(for example for the mouse imaging data) — or something at least close to that depending on how recent your version of Thunder is
then you should be able to load it with
data = tsc.loadImages(path)
path is that absolute path that gets you to the data on all of your nodes
data.first(), you get
(0, array([…])), but for a valid Thunder object, this should be
data = tsc.loadSeries(…).applyKeys(lambda k: (k,))