These are chat archives for thunder-project/thunder

24th
Jun 2015
aandreev
@aandreev0
Jun 24 2015 02:05

Hi! I installed thunder to run locally on Scientific Linux 7.1 through VirtualBox on 64bit Win7.
Python (Anaconda) v2.7.9, spark v1.4.0@hadoop1.x (downloaded binary). which thunder points in right place. However, when I launch thunder it throws error:

pyspark does not support any application options

Any ideas what happens and how to run thunder console? Thank you. (I'd use maillist but it looks dead...)

Jeremy Freeman
@freeman-lab
Jun 24 2015 02:08
@aandreev0 i believe the error you're describing was fixed in this commit: thunder-project/thunder@1f74209
it's a problem specific to Spark 1.4 (which was just release)
we'll try to push out an official patch that includes this fix (probably 0.5.1), but for now you can try either using the master branch of Thunder, or an earlier version of Spark (e.g. 1.3.1)
aandreev
@aandreev0
Jun 24 2015 03:07
@freeman-lab Thank you! Roll back to spark 1.3 solved it.
climfib
@climfib
Jun 24 2015 09:28
I installed thunder per (http://thunder-project.org/thunder/docs/install_local.html) instructions . Loading the example images works, but invoking image() does not bring up a window. any suggestions?
aandreev
@aandreev0
Jun 24 2015 10:37
@climfib post your code (all of it) snippet via https://gist.github.com or something else
Joel Kronborg
@Abobie
Jun 24 2015 10:39
Hello! I've installed thunder on 64-bit Windows 7, but I can't seem to launch it. When I type 'thunder' into cmd it just says that "'thunder' is not recognized as an internal or external command, operable program or batch file." I can import thunder in a Python console though, so it seems to be installed correctly. I'm using Python v2.7.9, spark 1.1.0. Any ideas what might be wrong?
Jason Wittenbach
@jwittenbach
Jun 24 2015 14:41
@climfib if you're running in the IPython terminal, then you probably need to set up matplotlib by adding the line %matplotlib. Of course, all of this will only work if you're running locally. If you're running on a server, then you probably need to look into using the IPython notebook for plotting.
climfib
@climfib
Jun 24 2015 15:03
@jwittenbach @aandreev0 thanks.
andrew giessel
@andrewgiessel
Jun 24 2015 17:12
just submitted : thunder-project/thunder#198
this fixes thunder-ec2 for spark 1.4.0
BUT WAIT THERE’S MORE
private_ips
omg git blame you are my hero
Jeremy Freeman
@freeman-lab
Jun 24 2015 17:16
@Abobie thanks for reporting, unfortunately this is a known problem and we just haven't gotten around to fixing it yet see herethunder-project/thunder#127
that issue offers a workaround, proper fix hopefully coming soon!
andrew giessel
@andrewgiessel
Jun 24 2015 18:18
@freeman-lab that PR is good to go, am running thunder on ec2 right now =)
if they could all be that simple...
Jeremy Freeman
@freeman-lab
Jun 24 2015 19:22
@andrewgiessel awesome! looking at it now, seems great
andrew giessel
@andrewgiessel
Jun 24 2015 19:41
quick question: if i want more modules or repos on a thunder-ec2 launched cluster, what should I do?
Jeremy Freeman
@freeman-lab
Jun 24 2015 19:49
to put something on the master just pip install mylibrary
and to put on the workers it's just pssh -h /root/spark-ec2/slaves 'source ~/.bash_profile && pip install mylibrary'
that should be enough
andrew giessel
@andrewgiessel
Jun 24 2015 19:49
in general, i need anything in a lambda on a slave?
Jeremy Freeman
@freeman-lab
Jun 24 2015 19:49
yeah that's the idea
andrew giessel
@andrewgiessel
Jun 24 2015 19:50
got it
thanks!
Jeremy Freeman
@freeman-lab
Jun 24 2015 19:50
if you're talking about custom libraries, you'll want to look at sc.addPyFile
which let's you ship an egg around to all the workers
usually you'd use that to send your own custom code around
we do that in thunder during startup
andrew giessel
@andrewgiessel
Jun 24 2015 19:50
can you use pip with a private repo?
that would be awesome.
andrew giessel
@andrewgiessel
Jun 24 2015 19:52
that is pretty awesome.
fun times.