These are chat archives for thunder-project/thunder

26th
Aug 2015
sophie63
@sophie63
Aug 26 2015 15:38
Hello everyone. I am blocked on accessing EC2 with thunder-ec2. I have been consistently getting a pssh error at the Anaconda installation step (see below). I have restarted using --resume many times, redid the AWS credentials and key setup steps and also freshly pip installed thunder-python --upgrade but still get the error.
'''
[Generating cluster's SSH key on master]
[success]
[Transferring cluster's SSH key to slaves]
[success]
[Deploying files to master]
[success]
[Installing Spark (may take several minutes)]
[success]
[Downloading Anaconda]
[success]
[Installing Anaconda]
[SSH failure, returning error]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/thunder/utils/ec2.py", line 514, in <module>
install_anaconda(master, opts)
File "/usr/local/lib/python2.7/dist-packages/thunder/utils/ec2.py", line 172, in install_anaconda
ssh(master, opts, "pssh -h /root/spark-ec2/slaves 'echo 'export "
File "/usr/local/lib/python2.7/dist-packages/thunder/utils/ec2.py", line 295, in ssh
raise Exception(stdout)
Exception: bash: pssh: command not found
'''
Did someone have a similar problem? Is there something I can try to fix that?
Jeremy Freeman
@freeman-lab
Aug 26 2015 15:39
hey @sophie63 , that indeed looks strange, can you open an issue on GitHub with this and I’ll look into it? thanks!
sophie63
@sophie63
Aug 26 2015 15:40
OK
Jeremy Freeman
@freeman-lab
Aug 26 2015 15:40
also, FYI, make sure to double check that this didn’t leave you with any instances running on EC2! log into your AWS console and shut them down manually if you need
sophie63
@sophie63
Aug 26 2015 15:40
ok thanks
Jeremy Freeman
@freeman-lab
Aug 26 2015 15:49
oh and maybe add what version of Spark you’re running?
sophie63
@sophie63
Aug 26 2015 16:08

does that make sense?

sc.version
u'1.1.0'

sophie63
@sophie63
Aug 26 2015 16:17
OK so I am trying to install a newer version of spark now
Jeremy Freeman
@freeman-lab
Aug 26 2015 16:17
ok, that might be worth trying, 1.1.0 is actually from awhile ago now!
let us know how it goes
it might fix it
sophie63
@sophie63
Aug 26 2015 17:13
:) I installed spark-1.4.1 and it fixed it.
I am closing the issue then.
Thanks!
Jeremy Freeman
@freeman-lab
Aug 26 2015 17:14
oh good to know! maybe leave it open and add a note to the ticket that we should update the documentation to say 1.4 or higher recommended for thunder-ec2
or whichever earlier version still works… it’s a little weird to me, because i thought it should have worked back that far
but glad you’re all set!
sophie63
@sophie63
Aug 26 2015 18:49
Just one last thing: when trying to use the notebook after doing
setup-notebook-ec2
source /root/.bash_profile
and checking that ipython-ops="notebook" was in the .bash_profile,
I still got the error:
ImportError: No module named notebook.notebookapp
but things work if I install jupyter (pip install jupyter)
andrew giessel
@andrewgiessel
Aug 26 2015 18:49
@sophie63 i had the same issue
I need to make an issue and dig into it
Jason Wittenbach
@jwittenbach
Aug 26 2015 19:03
Could be that the issue is caused by the somewhat recent split between the Jupyter front end and the IPython backend. It looks like the setup script gets IPython with the Anaconda install. I’m guessing that this used to bring the notebook front-end with it, but it probably no longer does and and the Jupyter notebook front end needs to be installed separately.
Probably don’t want to put Jupyter in the general requirements, but the ec2 setup script could install it separately
andrew giessel
@andrewgiessel
Aug 26 2015 19:10
@jwittenbach I think that is exactly correct
Jason Wittenbach
@jwittenbach
Aug 26 2015 19:10
On second thought, I guess the best place to put the Jupyter install would be in the setup-notebook-ec2 script =)
andrew giessel
@andrewgiessel
Aug 26 2015 19:11
Conda install Jupyter should work fibe
I think it is a collection of stuff
That's what I used to get it going. I can do a quick pr after I grab a slice
Jeremy Freeman
@freeman-lab
Aug 26 2015 19:13
Sounds cool guys, I'd probably put it in the main startup script just cause that already long and most people will want it
andrew giessel
@andrewgiessel
Aug 26 2015 19:38
thunder-project/thunder#232