These are chat archives for thunder-project/thunder

26th
Apr 2015
Tobias Rose
@trose-neuro
Apr 26 2015 08:58
sorry to bother you with EC2 issues again (it could be another RTFM thing...):
Now I can launch a cluster but I get the following SSH error (I tried multiple ---resume calls over the last hour, so I think it's not an issue of slow instance bootup):
    [Generating cluster's SSH key on master]
    [SSH failure, returning error]
Traceback (most recent call last):
  File "/Users/trose/anaconda/lib/python2.7/site-packages/thunder/utils/ec2.py", line 496, in <module>
    setup_cluster(conn, master_nodes, slave_nodes, opts, True)
  File "/Users/trose/anaconda/lib/python2.7/site-packages/thunder/utils/ec2.py", line 308, in setup_cluster
    ssh(master, opts, key_setup)
  File "/Users/trose/anaconda/lib/python2.7/site-packages/thunder/utils/ec2.py", line 287, in ssh
    raise Exception(stdout)
Exception
Jeremy Freeman
@freeman-lab
Apr 26 2015 14:12
hmmm, that one looks legit weird =)
did you chmod 600 you key file?
maybe you forgot after creating a new one?
Jeremy Freeman
@freeman-lab
Apr 26 2015 14:17
i'm trying to create this error by messing with the permissions on a key file and see if it gets stuck in the same place (and if so, will work on improving this uninformative error message!)
@josephwinston cool, yes definitely sounds like Series makes sense. so each "record" in the Series will be a one-dimensional array, and they all share a common index (in the simplest case 0 to n). and then maybe say again what is the kind of indexing you want to do?
Jeremy Freeman
@freeman-lab
Apr 26 2015 14:23
@ybot69 nevermind, an issue with key permissions should cause a failure earlier (before you get to "Generating cluster's SSH key on master")
Jeremy Freeman
@freeman-lab
Apr 26 2015 14:35
if you go to the EC2 console and find the IP address of the master, and just execute the following ssh -i ~/mykey.pem root@XX.XXX.XX.XX, does it work?
where XX.XX... is the IP address, as listed under the "Public IP" column on the EC2 console page
Richard A Hofer
@rhofour
Apr 26 2015 16:19
Ooh, our LU-decomposition looks like it finally works.
Going from here to an inverse should be pretty easy.
Tobias Rose
@trose-neuro
Apr 26 2015 16:48
@freeman-lab : no. I get:
ssh: connect to host 52.5.XX.XXX port 22: Connection refused
Tobias Rose
@trose-neuro
Apr 26 2015 16:54
@freeman-lab: Ok. Now after running ssh -i ~/mykey.pem root@XX.XXX.XX.XX again, I at least get the following error:
Permission denied (publickey).
I should probably start over and retrace my steps...
Jeremy Freeman
@freeman-lab
Apr 26 2015 17:16
@ybot69 are you on a Windows machine or Mac? and what version of Spark did you download?
Tobias Rose
@trose-neuro
Apr 26 2015 18:38
@freeman-lab MacOS 10.10.3, spark-1.1.0-bin-hadoop1. After launching a cluster I get the following Spark AMI: ami-35b1885c. The local thunder tutorials you've posted run just fine.
Jeremy Freeman
@freeman-lab
Apr 26 2015 18:40
not 100% sure it'll help but maybe try grabbing a more recent release of Spark, from here, try 1.3.0 (1.3.1 should work fine too, though we haven't tested against that yet)
Tobias Rose
@trose-neuro
Apr 26 2015 18:40
Will do. Thanks!
Still the built for Hadoop 1.X?
Jeremy Freeman
@freeman-lab
Apr 26 2015 18:42
yup
Tobias Rose
@trose-neuro
Apr 26 2015 18:43
loading. Will keep you posted.
Tobias Rose
@trose-neuro
Apr 26 2015 19:02

no... more info now, but same bottom line:

Warning: SSH connection error. (This could be temporary.)
Host: ec2-52-6-87-237.compute-1.amazonaws.com
SSH return code: 255
SSH output: Warning: Permanently added 'ec2-52-6-XX-XXX.compute-1.amazonaws.com,52.6.XX.XX' (RSA) to the list of known hosts.
Permission denied (publickey).

As I said: I'll try to go through the ec2 setup again. Maybe I did something wrong there.

Jeremy Freeman
@freeman-lab
Apr 26 2015 19:03
hm, this error should be due to an incorrect key-pair or a key-pair file with the wrong permissions