These are chat archives for ipython/ipython

10th
Nov 2015
zhoubug
@zhoubug
Nov 10 2015 06:34
i want to do some like quantopian.com ipython notebook style singe cell. how to config
cel4
@cel4
Nov 10 2015 07:53
Suppose I have a access to a cluster, but at the moment I am not so much interested in parallel computing, but in the power of a single node. So basically it would suffice to run a kernel on one of the nodes.
Can I do this with ipyparallel, or rather connection to a remote kernel?
cel4
@cel4
Nov 10 2015 09:06
More concrete question: I am running ipython on my notebook. I have access to the cluster via a "head node" in the cluster. Can I tell my ipython, that jobs can not be started here, but must be started from the head node?
Alejandro Guerrero Gonzalez
@aggFTW
Nov 10 2015 09:08
@cel4 What is this cluster?
cel4
@cel4
Nov 10 2015 09:09
common HPC with LSF batch system
cel4
@cel4
Nov 10 2015 09:19
I think I am having trouble understanding who is responsible for what in this case. Do I start the controller on the head node via ssh and the engines go via bsub in the cluster?
Min RK
@minrk
Nov 10 2015 09:55
That's a common way to do it, if you are allowed. Some systems will require that you run the controller on a work node as well.
It's definitely easier in most cases to run the controller on login.
cel4
@cel4
Nov 10 2015 11:53

In http://ipyparallel.readthedocs.org/en/latest/process.html?highlight=template#using-ipcluster-in-pbs-mode you write: Once you have created these scripts, save them with names like pbs.engine.template. Now you can load them into the ipcluster_config with [...]

Is there a reason for this dot style file name and where do you actually store these files?

Min RK
@minrk
Nov 10 2015 11:56
No, the name is irrelevant.
epifanio
@epifanio
Nov 10 2015 13:51
Hi, I’m having trouble with nbviewer. Same gist render fine on github but gives a 503 error on nbviewer
example gist: bf2ce5337657763a5f66 https://gist.github.com/anonymous/bf2ce5337657763a5f66
cel4
@cel4
Nov 10 2015 14:07

Just because I configured the controller to be on the login node does not mean that my LSF engines are launched from the controller host, does it?

Is there a config that tells ipython: "Whenever I want to start new engines via bsub, please ssh into the login node first and run bsub commands there"?

Min RK
@minrk
Nov 10 2015 14:12
ipcluster is quite primitive. It's basically a wrapper for 'when you want to start engines, run this command'
You use IPClusterEngines.engine_launcher_class to specify what is used to start engines. Starting the controller is configured separately, and the two have no effect on each other.
If the ipcluster command is being run on the login node, you probably want to use the LSF launcher.
If the ipcluster command (e.g. the notebook server) is being run somewhere else, there is an option to add an ssh hop, by using SSHProxy.
What SSHProxy does is say "on this computer, run the command ssh <HOST> ipcluster engines -n <N>"
In which case you would configure the local machine to use SSHProxy, and make an IPython profile on the login node that tells it to use LSF.
cel4
@cel4
Nov 10 2015 14:43
So basically I configure the computer with the notebook frontend to use a SSHProxyEngineSetLauncher, right? Are there docs what to write in the config files for that?
cel4
@cel4
Nov 10 2015 14:49
I mean apart from the obvious c.IPClusterEngines.engine_launcher_class = 'SSHProxyEngineSetLauncher'
Min RK
@minrk
Nov 10 2015 15:09
I don't think there are thorough docs of using SSHProxy, but the gist is:
  1. [login node]$ ipython profile create --parallel
  2. [login node] configure default profile to use LSF (c.IPClusterEngines.engine_launcher_class = 'lsf')
  3. [laptop/etc.]$ ipython profile create --parallel
  4. [laptop/etc.] configure profile to use SSHProxy, set SSHProxyEngineSetLauncher.hostname = 'cluster_host'
epifanio
@epifanio
Nov 10 2015 15:43
same link it is now working .. on nbviewer
cel4
@cel4
Nov 10 2015 16:12
Hmh, seems like I managed to configure all the steps. SSHProxyEngineSetLauncher connects successfully via ssh to the login node, the login node then starts theLSFEngineSetLauncher. Unfortunately, at this point things go awry. Despite of being on the node, I see [D 16:56:56.983 NotebookApp] b'/bin/sh: bsub: command not found\r' in the debug log. Is there any chance that I can give the full path to bsub to see if there's a sourcing issue?
Min RK
@minrk
Nov 10 2015 16:20
You can, I'm not 100% sure what the name is, probably batch_cmd or similar.
cel4
@cel4
Nov 10 2015 16:46
Hmh, BatchSystemLauncher has the attribute submit_command, but setting c.PBSEngineSetLauncher.submit_command = bla on the login node has no effect as far as I can see.
Min RK
@minrk
Nov 10 2015 17:51
c.LSFEngineSetLauncher, if that's the one you are using.
cel4
@cel4
Nov 10 2015 18:23
argh thanks, stupid mistake... guess I should get some sleep...