These are chat archives for ipython/ipython

30th
Apr 2015
Bas Nijholt
@basnijholt
Apr 30 2015 16:02
I'm trying to start engines over SSH. The problem is that Python lives in a virtualenv and c.SSHEngineSetLauncher.engine_cmd = ['/path/to/your/python', '-c', 'from IPython.parallel.apps.ipengineapp import launch_new_instance; launch_new_instance()'] doesn't work. I keep getting the following error:
basnijholt:~> ipcluster start --profile=ssh
2015-04-30 17:59:26.740 [IPClusterStart] Starting ipcluster with [daemon=False]
2015-04-30 17:59:26.741 [IPClusterStart] Creating pid file: /Users/basnijholt/.ipython/profile_ssh/pid/ipcluster.pid
2015-04-30 17:59:26.742 [IPClusterStart] Starting Controller with LocalControllerLauncher
2015-04-30 17:59:27.749 [IPClusterStart] Starting 4 Engines with SSH
2015-04-30 17:59:30.758 [IPClusterStart] ensuring remote hpc05:.ipython/profile_ssh/security/ exists
Connection to hpc05.tudelft.net closed.
2015-04-30 17:59:31.095 [IPClusterStart] sending /Users/basnijholt/.ipython/profile_ssh/security/ipcontroller-client.json to hpc05:.ipython/profile_ssh/security/ipcontroller-client.json
2015-04-30 17:59:31.412 [IPClusterStart] ensuring remote hpc05:.ipython/profile_ssh/security/ exists
Connection to hpc05.tudelft.net closed.
2015-04-30 17:59:31.678 [IPClusterStart] sending /Users/basnijholt/.ipython/profile_ssh/security/ipcontroller-engine.json to hpc05:.ipython/profile_ssh/security/ipcontroller-engine.json
ssh: connect to host hpc05.tudelft.net port 22: Operation timed out
lost connection
2015-04-30 17:59:39.546 [IPClusterStart] ERROR | Engine start failed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/ipclusterapp.py", line 332, in start_engines
    self.engine_launcher.start(self.n)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 779, in start
    d = el.start(user=user, hostname=host)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 634, in start
    self.send_files()
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 602, in send_files
    self._send_file(local_file, remote_file)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 595, in _send_file
    check_output(self.scp_cmd + [local, full_remote])
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['scp', u'/Users/basnijholt/.ipython/profile_ssh/security/ipcontroller-engine.json', u'hpc05:.ipython/profile_ssh/security/ipcontroller-engine.json']' returned non-zero exit status 1
ERROR:tornado.application:Exception in callback <functools.partial object at 0x101fc3b50>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/tornado/ioloop.py", line 568, in _run_callback
    ret = callback()
  File "/usr/local/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/ipclusterapp.py", line 332, in start_engines
    self.engine_launcher.start(self.n)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 779, in start
    d = el.start(user=user, hostname=host)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 634, in start
    self.send_files()
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 602, in send_files
    self._send_file(local_file, remote_file)
  File "/usr/local/lib/python2.7/site-packages/IPython/parallel/apps/launcher.py", line 595, in _send_file
    check_output(self.scp_cmd + [local, full_remote])
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['scp', u'/Users/basnijholt/.ipython/profile_ssh/security/ipcontroller-engine.json', u'hpc05:.ipython/profile_ssh/security/ipcontroller-engine.json']' returned non-zero exit status 1
Anton Akhmerov
@akhmerov
Apr 30 2015 16:14
@basnijholt Can you actually try running the last command with some mock file and checking the error message?
Doesnt' seem related to the virtual environment.
Bas Nijholt
@basnijholt
Apr 30 2015 16:24
The strange thing is that ipcontroller-client.json is actually copied.
Anton Akhmerov
@akhmerov
Apr 30 2015 16:28
But it's the engine.json that fails to be copied.
Bas Nijholt
@basnijholt
Apr 30 2015 16:44
True. But when I try to run the command, it first will work, but when the ipcluster is also doing it, it fails. Also tried it on a different machine, with same profile and get the same problem.
Min RK
@minrk
Apr 30 2015 16:51
When you run the scp command manually it works?
Bas Nijholt
@basnijholt
Apr 30 2015 17:09
Not at the moment when ipcluster tries to copy it
But just before that it works fine.
Dale Jung
@dalejung
Apr 30 2015 20:57
I had to comment out the jupyter_notebook req in nbconvert's setup.py. since they both reference each other.
prevented me from installing both
am I doing something boneheaded wrong?
Min RK
@minrk
Apr 30 2015 21:00
@dalejung no, it's a circular dependency, which is troublesome while nothing has been released, yet. pip install -r requirements -e . should work.
The key is installing both with the same command, so pip knows both dependencies are satisfied.
Dale Jung
@dalejung
Apr 30 2015 21:01
ah gotcha. I was doing it pgk by pkg. good to know
Bas Nijholt
@basnijholt
Apr 30 2015 21:15
@minrk no idea where this error comes from?
Kyle Kelley
@rgbkrk
Apr 30 2015 21:53
Frameless
Screenshot 2015-04-30 16.53.26.png
Damian Avila
@damianavila
Apr 30 2015 21:56
my skype is busted... and I have another meeting in5 min... have fun!!
Damian Avila
@damianavila
Apr 30 2015 22:35
/@all ^^^, btw, I just saw the summary, it seems I don't have to buy any beer to ask for the name :wink2:
Kyle Kelley
@rgbkrk
Apr 30 2015 23:56
You should buy them a beer anway
plus, q is a better package for what they're using inspect for