These are chat archives for dereneaton/ipyrad

21st
Sep 2017
tommydevitt
@tommydevitt
Sep 21 2017 02:33
@isaacovercast @dereneaton I've tried the API both with and without specifying a profile and can't get the notebook to connect to the ipcluster instance. I assume I need to first create a profile before specifying it in the batch file? Is the ordering important when submitting the slurm_ipcluster_MPI.sbatch and slurm_jupyter.sbatch scripts?
Here's what I'm getting:
TimeoutErrorTraceback (most recent call last)
<ipython-input-11-a91977b5d2d2> in <module>()
      3 
      4 ## connect to the client
----> 5 ipyclient = ipp.Client()
      6 
      7 ## print how many engines are connected

/home1/02745/tdevitt/miniconda2/lib/python2.7/site-packages/ipyparallel/client/client.pyc in __init__(self, url_file, profile, profile_dir, ipython_dir, context, debug, sshserver, sshkey, password, paramiko, timeout, cluster_id, **extra_args)
    493 
    494         try:
--> 495             self._connect(sshserver, ssh_kwargs, timeout)
    496         except:
    497             self.close(linger=0)

/home1/02745/tdevitt/miniconda2/lib/python2.7/site-packages/ipyparallel/client/client.pyc in _connect(self, sshserver, ssh_kwargs, timeout)
    613         evts = poller.poll(timeout*1000)
    614         if not evts:
--> 615             raise error.TimeoutError("Hub connection request timed out")
    616         idents, msg = self.session.recv(self._query_socket, mode=0)
    617         if self.debug:

TimeoutError: Hub connection request timed out
Amanda Haponski
@ahaponski_twitter
Sep 21 2017 13:50
Hi @dereneaton and @isaacovercast!!! I have been trying to run the ABBA-BABA tests based on the cookbook available on Github (cookbook-abba-baba.ipynb). Everything loads and runs correctly until I get to the bb.run(ipyclient) cell. I keep getting the error below. Any ideas? Thanks in advance!!!
[                    ]   0%  calculating D-stats  | 0:00:06 |  
---------------------------------------------------------------------------
IPyradError                               Traceback (most recent call last)
<ipython-input-10-31a33f8414f6> in <module>()
----> 1 aa.run(ipyclient)

/Users/Amanda/miniconda2/lib/python2.7/site-packages/ipyrad/analysis/baba.pyc in run(self, ipyclient)
    120             An ipyparallel client object to distribute jobs to a cluster.
    121         """
--> 122         self.results_table, self.results_boots = batch(self, ipyclient)
    123         self.results_table.nloci = np.nan_to_num(self.results_table.nloci)\
    124                                                  .astype(int)

/Users/Amanda/miniconda2/lib/python2.7/site-packages/ipyrad/analysis/baba.pyc in batch(baba, ipyclient)
    276                 if not asyncs[job].successful():
    277                     raise IPyradWarningExit(\
--> 278                         " error: {}: {}".format(job, asyncs[job].exception()))
    279                 ## enter results for successful jobs
    280                 else:

/Users/Amanda/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/util.pyc in __init__(self, *args, **kwargs)
     50     def __init__(self, *args, **kwargs):
     51         if ipyrad.__interactive__:
---> 52             raise IPyradError(*args, **kwargs)
     53         else:
     54             SystemExit.__init__(self, *args, **kwargs)

IPyradError:  error: 0: ValueError(cannot assign slice from input of different size)
Deren Eaton
@dereneaton
Sep 21 2017 17:48
Hi @tommydevitt, you got it to work when run without MPI, right? When you run the ipcluster submission script with the MPI flag does it report any errors? If it works without MPI but not with it then it might be that ipcluster shuts down when it is not able to establish the MPI connection, which would explain why you can't connect to it.
tommydevitt
@tommydevitt
Sep 21 2017 18:03
@dereneaton No, I don't get any errors. The log file indicates that the engines start successfully:
2017-09-20 21:21:15.387 [IPClusterStart] Removing pid file: /home1/02745/tdevitt/.ipython/profile_default/pid/ipcluster.pid
2017-09-20 21:21:15.388 [IPClusterStart] Starting ipcluster with [daemon=False]
2017-09-20 21:21:15.390 [IPClusterStart] Creating pid file: /home1/02745/tdevitt/.ipython/profile_default/pid/ipcluster.pid
2017-09-20 21:21:15.390 [IPClusterStart] Starting Controller with LocalControllerLauncher
2017-09-20 21:21:16.395 [IPClusterStart] Starting 96 Engines with MPI
2017-09-20 21:21:46.401 [IPClusterStart] Engines appear to have started successfully
Deren Eaton
@dereneaton
Sep 21 2017 18:04
and if you call ipp.Client() with no profile argument now it still doesn't find it?
If that doesn't work, try passing it in the location of the controller file directly:
ipp.Client(url_file="/home1/02745/tdevitt/.ipython/profile_default/security/ipcontroller-client.json")
tommydevitt
@tommydevitt
Sep 21 2017 18:07
Right.
OK, let me try that.
tommydevitt
@tommydevitt
Sep 21 2017 19:57

@dereneaton I'm still waiting to connect to a compute node on my HPC...

Related to this problem (that is, having to submit two slurm jobs, at least one of which I have to wait in a queue for) is it possible to run the most computationally intensive steps of the pipeline non-interactively and use the API starting somewhere in the middle of the pipeline for only the steps that require more interaction (e.g., branching, creating various combinations of parameters for an analysis, etc.)? In other words, I'd like to just run step 3 via a regular batch script and use the API subsequently for the steps that require more tinkering, starting at a later step. Am I making sense?

Maybe I'm not thinking about this in the right way, but submitting one job to run the jupyter notebook and a second job to run the actual computations leaves me waiting for the moment when both jobs have started before I can do anything. Am I missing something?

Deren Eaton
@dereneaton
Sep 21 2017 20:25

@tommydevitt yeah you can jump between the API and CLI quite easily. For example, if you started in the CLI then you can switch to the API at any point by simply loading the JSON representation of the Assembly object in jupyter/ipythong with the following command:

import ipyrad as ip
data = ip.load_json("my_assembly.json")

Then if after running a few operations on it using the API you want to switch back to the CLI you could write a new params-file for any Assembly by using the the following function:

data.write_params()

If you only want to use the API to do branching and changing parameters and stuff then you actually don't need to have ipcluster running at all. The only time ipyrad uses the parallelization with ipyparallel is during a .run() command.

tommydevitt
@tommydevitt
Sep 21 2017 20:27
@dereneaton OK, that's very helpful, thanks Deren.