These are chat archives for dereneaton/ipyrad

16th
Aug 2017
Bohao Fang
@fangbohao_twitter
Aug 16 2017 13:39
Hi @dereneaton, thank you for the upgrade of BUCKy! Old problems are fixed, but "b.run_mbsum(ipyclient=ipyclient)" has a problem of "stdout". Please see following screen shot:
ScreenClip.png
Deren Eaton
@dereneaton
Aug 16 2017 15:01
Hi @fangbohao_twitter , oh, dang. I'll make a fix for that. That error will only arise when it is trying to print the reason why an earlier error occurred during the mbsum command, so there must be a problem earlier as well, which is what we really want to figure out. I haven't spent much time yet on writing stuff into the ipa.bucky code to catch errors and report them cleanly. Try running your data using the single .run(ipyclient=ipyclient, force=True) command, which calls each of the other three subcommands within it to run mrbayes, mbsum, and bucky in order. I hope we can get it working for you soon.
Bohao Fang
@fangbohao_twitter
Aug 16 2017 15:30
Hi @dereneaton, I tried "b.run(ipyclient=ipyclient, force=True)", the problem is the same. Anyway, thank you very much for your efforts!
ScreenClip.png
Deren Eaton
@dereneaton
Aug 16 2017 15:38
OK, well it seems to be running the first step ok, so that's good.
did you check that mbsum and bucky were installed? They should be callable from the command line if you installed them from the ipyrad conda channel with conda install bucky -c ipyrad
toczydlowski
@toczydlowski
Aug 16 2017 15:46
@dereneaton Yes! That makes a ton of sense, and is something that's bugged/confused me for awhile. Thanks!! Always appreciate you taking the time to engage with users.
tommydevitt
@tommydevitt
Aug 16 2017 18:37
Hi @dereneaton , I'm getting the following error after updating conda and ipyrad:
IOError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.
Deren Eaton
@dereneaton
Aug 16 2017 18:39
@tommydevitt are you using the ipyrad API when you get this error?
tommydevitt
@tommydevitt
Aug 16 2017 18:41
@dereneaton no, just the CLI
I seem to remember @edgardomortiz_twitter having similar trouble about a year ago. We're using the same HPC..
Deren Eaton
@dereneaton
Aug 16 2017 18:42
Aha, but you haven't seen this problem before on that HPC?
yourself
it just means that it's taking a little too long to find the ipcluster instance, which is a tool ipyrad uses to connect many cores on a cluster.
tommydevitt
@tommydevitt
Aug 16 2017 18:44
I seem to remember having the same problem, but we resolved it previously doing ipcluster start --n 24 --profile=ipyrad --daemonize followed by a sleep command
Deren Eaton
@dereneaton
Aug 16 2017 18:44
This can happen on really big or slow filesystems
tommydevitt
@tommydevitt
Aug 16 2017 18:45
so should I increase the time after the sleep command?
Deren Eaton
@dereneaton
Aug 16 2017 18:45
yes, that would work. But we changed how it works just a bit now. We no longer use --profile=ipyrad as the default profile, but instead let you set the name to whatever you want.
You just need to set the same profile for ipcluster that you do for ipyrad, like this:
ipcluster start --n=24 --profile=watwat
sleep 30
ipyrad -p params.txt --ipcluster watwat
where 'watwat' can be whatever you want.
tommydevitt
@tommydevitt
Aug 16 2017 18:46
I'll give it a try
Bohao Fang
@fangbohao_twitter
Aug 16 2017 18:46
Hi @dereneaton , thank you for your reply. I check that 'mbsum' and 'bucky' are already installed and "return stdout" problem is here when rerun the BUCKy again.
tommydevitt
@tommydevitt
Aug 16 2017 18:55
seems to have hung after the engines started: nid00008(1)$ ipcluster start --n 48 --profile=test
2017-08-16 13:48:02.571 [IPClusterStart] Created profile dir: u'/home1/02745/tdevitt/.ipython/profile_test'
2017-08-16 13:48:02.589 [IPClusterStart] Starting ipcluster with [daemon=False]
2017-08-16 13:48:02.591 [IPClusterStart] Creating pid file: /home1/02745/tdevitt/.ipython/profile_test/pid/ipcluster.pid
2017-08-16 13:48:02.592 [IPClusterStart] Starting Controller with LocalControllerLauncher
2017-08-16 13:48:03.598 [IPClusterStart] Starting 48 Engines with LocalEngineSetLauncher
2017-08-16 13:48:39.708 [IPClusterStart] Engines appear to have started successfully
Deren Eaton
@dereneaton
Aug 16 2017 18:58
Oh yeah, you either need to --daemonize it, or run it in a separate terminal. It is essentially running a server, so you need to keep it running as long as you wish to connect to it. Normally this is what ipyrad does behind the scenes, it starts it up, runs a bunch of work, and then shuts it down.
Deren Eaton
@dereneaton
Aug 16 2017 19:04
You'll probably want to just add the --daemonize flag if using the CLI.
tommydevitt
@tommydevitt
Aug 16 2017 19:04
success! thanks @dereneaton !
Deren Eaton
@dereneaton
Aug 16 2017 19:05
The advanced usage of ipcluster is mostly meant for using the ipyrad API, where you can do a bunch of cool stuff in jupyter notebook connected to an ipcluster. There's a bunch of info, and a video about it here: http://ipyrad.readthedocs.io/HPC_Tunnel.html.
glad to hear its working!
tommydevitt
@tommydevitt
Aug 16 2017 19:07
I see. I've just started toying around with the jupyter notebook.
Deren Eaton
@dereneaton
Aug 16 2017 19:09
cool. It's definitely easier to start by learning to run it locally, but once you get that down and figure out how to run it on a remote cluster it's super cool.
tommydevitt
@tommydevitt
Aug 16 2017 19:11
I've got it running locally no problem; trying to understand how to run it most efficiently via my HPC; it
does seem very elegant
tommydevitt
@tommydevitt
Aug 16 2017 21:49
@dereneaton : one small typo on the pedicularis API tutorial: in the first cell, sratools should be sra-tools, I think.