These are chat archives for dereneaton/ipyrad

24th
Aug 2016
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:02

Hello, I am getting the following error after updating to 0.3.32 I tried uninstalling conda completely and reinstalling with no luck:


ipyrad [v.0.3.32]

Interactive assembly and analysis of RAD-seq data

New Assembly: ast

No ipcluster instance found. This may be a problem with your installation
setup. I would recommend that you contact the ipyrad developers.
Deren Eaton
@dereneaton
Aug 24 2016 15:04
@edgardomortiz_twitter Is this on HPC?
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:06
Deren Eaton
@dereneaton
Aug 24 2016 15:10
ok, try updating to 0.3.33 now and running it.
it's working for me, but I'm not positive that will fix the problem... let me know.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:13
Hi again, same error after updating...
Deren Eaton
@dereneaton
Aug 24 2016 15:15
OK, try something simple for me, go into your .json file and find a value called "timeout" and increase it from 45 to 90. Then try running again.
I think that might fix it
or let me know if you don't see a value
hmm, maybe it won't be in there actually... sorry. I'll push an update that I think will work.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:20
yeah, that value is not in the .json
Deren Eaton
@dereneaton
Aug 24 2016 15:22
OK, updating should give you v.0.3.33-2. Give it a try.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:29
oops, same error, can't find timeout in the new .json either
Deren Eaton
@dereneaton
Aug 24 2016 15:30
That's alright, 'timeout' won't be in it. The default value in the code (now 120 instead of 45) should hopefully work.
but I guess not.
hmm ...
can you open an interactive session?
I think we can figure out the problem then.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:50
I just ran a "live" session on a single node (48 cores) and it showed me the same error
Deren Eaton
@dereneaton
Aug 24 2016 15:56
are you logged into the node now so that you can run some code?
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 15:58
yes
Deren Eaton
@dereneaton
Aug 24 2016 15:59
great, try the following::
## in the terminal start an ipcluster instance
$ ipcluster start --n 4
what does it print?
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:00
This so far:
2016-08-24 10:59:56.759 [IPClusterStart] Starting ipcluster with [daemon=False]
2016-08-24 10:59:56.762 [IPClusterStart] Creating pid file: /home1/02728/emo347/.ipython/profile_default/pid/ipcluster.pid
2016-08-24 10:59:56.762 [IPClusterStart] Starting Controller with LocalControllerLauncher
2016-08-24 10:59:57.770 [IPClusterStart] Starting 4 Engines with LocalEngineSetLauncher
Deren Eaton
@dereneaton
Aug 24 2016 16:01
copy the last line when it arrives, it should say "Engines appear to have started successfully"
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:01
2016-08-24 11:00:28.119 [IPClusterStart] Engines appear to have started successfully
Deren Eaton
@dereneaton
Aug 24 2016 16:02
OK, hmm, so it took about 30 seconds.
use ctrl-c to stop the ipcluster
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:02
done
Deren Eaton
@dereneaton
Aug 24 2016 16:02
then run this:
ipcluster start --n 4 --daemonize
And then type ipython to open an IPython session
in ipython try the following:
import ipyparallel as ipp

rc = ipp.Client()
rc[:]
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:04
Out[1]: <DirectView [0, 1, 2, 3]>
Deren Eaton
@dereneaton
Aug 24 2016 16:04
Well, that's good that that's working.
try this:
import ipyparallel as ipp

rc = ipp.Client(profile="default")
rc[:]
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:05
Out[2]: <DirectView [0, 1, 2, 3]>
Deren Eaton
@dereneaton
Aug 24 2016 16:06
ok now:
import ipyrad as ip

## location of your json file here
data = ip.load_json("dir/path.json")

print data._ipcluster
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:09
I am recreating the .json, is taking a while
IPyradError:
    No samples found in saved assembly. If you are just starting a new
    assembly the file probably got saved erroneously, so it's safe to try
    removing the assembly file (e.g., rm ./ast.json.json) and restarting.

    If you fully completed step 1 and you see this message you should probably
    contact the developers.
Deren Eaton
@dereneaton
Aug 24 2016 16:12
did this error occur during the 'load_json' func?
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:13
yes, so datacan't be defined
Deren Eaton
@dereneaton
Aug 24 2016 16:14
let's try just starting from scratch. We have a blank Assembly (data) and we'll add the path to your data and/or barcodes:
## print params
data.get_params()

## set new parameters
data.set_params("raw_fastq_path", "path_to_data/*.gz")
data.set_params("barcodes_path", "path_to_barcode.txt")
whoops, sorry: start with the following first:
data = ip.Assembly('test')
For speed maybe just put the location to one of your data files in 'raw_fastq_path' or 'sorted_fastq_path'.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:17
done:
data.set_params("sorted_fastq_path", "./ast_fastqs/*.gz")
Deren Eaton
@dereneaton
Aug 24 2016 16:17
ok now run:
data.run('1')
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:19
I didn't change the datatype to paired, should I?
  Assembly: test

    Linking to demultiplexed fastq files in: /scratch/02728/emo347/ddRAD/diplo/e_pseudoreference_ipyrad0332/ast_fastqs/*.gz
    Warning: '_R2_' was detected in a file name, which suggests the data may
    be paired-end. If so, you should set the parameter 'datatype' to a paired
    option (e.g., pairddrad or pairgbs) and re-run step 1, which will require
    using the force flag (-f) to overwrite existing data.
Deren Eaton
@dereneaton
Aug 24 2016 16:19
I suppose, but it's just a warning, it should still run.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:20
Yes, it is running...
Deren Eaton
@dereneaton
Aug 24 2016 16:21
when it is done try the following:
print data.stats
print data._ipcluster
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:22
               state  reads_raw
AS1-Aaspe          1     556641
AS1-Aaspe_R2_      1     556641
AS1-Bbart          1     104423
AS1-Bbart_R2_      1     104423
AS2-Amatu          1     863100
AS2-Amatu_R2_      1     863100
AS2-Bgeni          1    2470169
AS2-Bgeni_R2_      1    2470169
AS2-Btric          1    3003872
AS2-Btric_R2_      1    3003872
AS2-Fhyps          1     357655
AS2-Fhyps_R2_      1     357655
AS2-Halie          1    3221715
AS2-Halie_R2_      1    3221715
AS2-Pquad          1    2887763
AS2-Pquad_R2_      1    2887763
DIP-Dbarc          1     164278
DIP-Dbarc_R2_      1     164278
DIP-Dcall          1    1954837
DIP-Dcall_R2_      1    1954837
DIP-Dempe          1     632139
DIP-Dempe_R2_      1     632139
DIP-Deric          1    3244720
DIP-Deric_R2_      1    3244720
DIP-Dglan          1     441227
DIP-Dglan_R2_      1     441227
DIP-Dgnid          1      15172
DIP-Dgnid_R2_      1      15172
DIP-Dgood          1    2717676
DIP-Dgood_R2_      1    2717676
DIP-Dgyno          1     435726
DIP-Dgyno_R2_      1     435726
DIP-Dhaen          1     208286
DIP-Dhaen_R2_      1     208286
DIP-Dhart          1     110395
DIP-Dhart_R2_      1     110395
DIP-Doxap          1    1413800
DIP-Doxap_R2_      1    1413800
DIP-Dpulc          1     205259
DIP-Dpulc_R2_      1     205259
DIP-Dspin          1     181831
DIP-Dspin_R2_      1     181831
DIP-Dspj1          1    2279931
DIP-Dspj1_R2_      1    2279931
DIP-Dspj3          1    1660481
DIP-Dspj3_R2_      1    1660481
HYB-Dcine          1     232035
HYB-Dcine_R2_      1     232035
HYB-Dspc2          1    2701937
HYB-Dspc2_R2_      1    2701937
OUT-Operu          1     573858
OUT-Operu_R2_      1     573858
PIO-Dalve          1    3805356
PIO-Dalve_R2_      1    3805356
PIO-Danti          1     231195
PIO-Danti_R2_      1     231195
PIO-Dapic          1    2573653
PIO-Dapic_R2_      1    2573653
PIO-Dcaya          1    3106137
PIO-Dcaya_R2_      1    3106137
PIO-Dcine          1       5018
PIO-Dcine_R2_      1       5018
PIO-Dcolo          1     700913
PIO-Dcolo_R2_      1     700913
PIO-Derio          1    2721739
PIO-Derio_R2_      1    2721739
PIO-Dfron          1     196675
PIO-Dfron_R2_      1     196675
PIO-Dglut          1    1273058
PIO-Dglut_R2_      1    1273058
PIO-Djene          1     379127
PIO-Djene_R2_      1     379127
PIO-Dphyl          1    2121034
PIO-Dphyl_R2_      1    2121034
PIO-Drevo          1    1277399
PIO-Drevo_R2_      1    1277399
PIO-Drhom          1     566697
PIO-Drhom_R2_      1     566697
PIO-Drosm          1    1616403
PIO-Drosm_R2_      1    1616403
PIO-Drupe          1     208797
PIO-Drupe_R2_      1     208797
PIO-Dschu          1    1176875
PIO-Dschu_R2_      1    1176875
PIO-Dtenu          1     461298
PIO-Dtenu_R2_      1     461298
{'profile': 'default', 'engines': 'Local', 'quiet': 0, 'cluster_id': '', 'timeout': 120, 'cores': 48}
Deren Eaton
@dereneaton
Aug 24 2016 16:24
Now
data.write_params('params-test.txt')
and then exit
So the API is working fine, but for some reason the CLI doesn't seem to be able to get ipcluster running. Let's try it with this new assembly.
stop the ipcluster instance that is running in the background:
ipcluster stop
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:26
2016-08-24 11:26:06.968 [IPClusterStop] Stopping cluster [pid=18578] with [signal=2]
Deren Eaton
@dereneaton
Aug 24 2016 16:26
and then see if the ipyrad CLI can run with this new params file by launching it's own ipcluster instance:
ipyrad -p params-test.txt -s 1 -f
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:29

 -------------------------------------------------------------
  ipyrad [v.0.3.33]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  New Assembly: test

    No ipcluster instance found. This may be a problem with your installation
    setup. I would recommend that you contact the ipyrad developers.
weird
Deren Eaton
@dereneaton
Aug 24 2016 16:29
argh! Alright, well I'll mess around with it for a while and try to figure it out. Thanks, that was really helpful.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:31
Thanks to you, that was educational :)
Isaac Overcast
@isaacovercast
Aug 24 2016 16:35
@edgardomortiz_twitter Do you happen to know what OS the compute nodes use?
oops nvm, i see it in the docs you linked to: SUSE
There's another user having this exact problem, trying to identify commonalities
Deren Eaton
@dereneaton
Aug 24 2016 16:37
@isaacovercast It's strange that worked fine through the API but not the CLI?
maybe __main__.py is messing with something...
I have access to two clusters and I can't replicate it on either. You think it's specific to hardware?
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:41
just in case:
LSB Version:       core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID:    SUSE LINUX
Description:       SUSE Linux Enterprise Server 11 (x86_64)
Release:           11
Codename:          n/a
Isaac Overcast
@isaacovercast
Aug 24 2016 16:43
It is weird that API works and CLI doesn't. I don't think its hw specific, but probably some weird environment issue.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 16:47
Let me try submitting the job to a "normal" node, these tests were done in a "development" one
Nope, I was submitting to "normal" all the time, the interactive session was trough a "development" node
Deren Eaton
@dereneaton
Aug 24 2016 16:53
@edgardomortiz_twitter let's try one more thing:
## find the most recent ipcontroller log file (will be sorted to bottom of list)
ls -thr ~/.ipython/profile_default/log/

## copy us the printout of the last file in list with 'ipcontroller' in its name
cat ~/.ipython/profile_default/log/ipcontroller-ipyrad-{N}-{M}.log
you could do this from the login node or a compute node.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 17:06
part 1
2016-08-24 09:49:56.371 [IPControllerApp] Hub listening on tcp://127.0.0.1:25567 for registration.
2016-08-24 09:49:56.410 [IPControllerApp] Hub using DB backend: 'DictDB'
2016-08-24 09:49:56.741 [IPControllerApp] hub::created hub
2016-08-24 09:49:56.742 [IPControllerApp] writing connection info to /home1/02728/emo347/.ipython/profile_default/security/ipcontroller-ipyrad-32984-client.json
2016-08-24 09:49:56.776 [IPControllerApp] writing connection info to /home1/02728/emo347/.ipython/profile_default/security/ipcontroller-ipyrad-32984-engine.json
2016-08-24 09:49:56.807 [IPControllerApp] task::using Python leastload Task scheduler
2016-08-24 09:49:56.809 [IPControllerApp] Heartmonitor started
2016-08-24 09:49:57.018 [IPControllerApp] Creating pid file: /home1/02728/emo347/.ipython/profile_default/pid/ipcontroller-ipyrad-32984.pid
2016-08-24 09:49:57.248 [IPControllerApp] client::client '\x00k\x8bEg' requested u'connection_request'
2016-08-24 09:49:57.249 [IPControllerApp] client::client ['\x00k\x8bEg'] connected
2016-08-24 09:50:02.485 [IPControllerApp] client::client '05dcbde5-37c3-4796-8699-8a81fcf68202' requested u'registration_request'
2016-08-24 09:50:04.148 [IPControllerApp] client::client 'd05732d2-a649-423d-974c-e6f7b1d22d88' requested u'registration_request'
2016-08-24 09:50:04.314 [IPControllerApp] client::client '18c565fe-4684-4e32-8864-95e50616d2e5' requested u'registration_request'
2016-08-24 09:50:05.516 [IPControllerApp] client::client 'b1af984a-2fa3-4e3e-a147-bd4ad1ac7da3' requested u'registration_request'
2016-08-24 09:50:05.819 [IPControllerApp] client::client 'e843f060-c459-44d2-86b3-5e9c1ac59ec0' requested u'registration_request'
2016-08-24 09:50:06.907 [IPControllerApp] client::client '843ab55a-bd6c-495a-82c1-772c12c58cf0' requested u'registration_request'
2016-08-24 09:50:07.418 [IPControllerApp] client::client 'db91ec59-1d2e-4118-97e0-ecebd1c91502' requested u'registration_request'
2016-08-24 09:50:07.724 [IPControllerApp] client::client 'd0dd2841-1084-4caf-9d6c-6b3f9a1d214e' requested u'registration_request'
2016-08-24 09:50:08.400 [IPControllerApp] client::client 'd4346b48-c2df-42c0-b35c-afee186a4d24' requested u'registration_request'
2016-08-24 09:50:08.809 [IPControllerApp] registration::finished registering engine 0:05dcbde5-37c3-4796-8699-8a81fcf68202
part 2
2016-08-24 09:50:08.810 [IPControllerApp] engine::Engine Connected: 0
2016-08-24 09:50:08.818 [IPControllerApp] registration::finished registering engine 1:d05732d2-a649-423d-974c-e6f7b1d22d88
2016-08-24 09:50:08.819 [IPControllerApp] engine::Engine Connected: 1
2016-08-24 09:50:08.829 [IPControllerApp] registration::finished registering engine 2:18c565fe-4684-4e32-8864-95e50616d2e5
2016-08-24 09:50:08.833 [IPControllerApp] engine::Engine Connected: 2
2016-08-24 09:50:08.866 [IPControllerApp] client::client '980fea61-7d2c-4710-abbf-5640da24f9e2' requested u'registration_request'
2016-08-24 09:50:08.935 [IPControllerApp] client::client '9810f9ea-fcac-485f-923d-70200142c0d5' requested u'registration_request'
2016-08-24 09:50:09.741 [IPControllerApp] client::client '3764babd-e59b-48ba-b6ac-73106d9ac663' requested u'registration_request'
2016-08-24 09:50:10.165 [IPControllerApp] client::client '3a0a81a1-a499-4c12-8166-c2406af92398' requested u'registration_request'
2016-08-24 09:50:10.298 [IPControllerApp] client::client '4745edf2-df44-4971-b812-5755d285b1f7' requested u'registration_request'
2016-08-24 09:50:10.844 [IPControllerApp] client::client 'ad652bf1-cc72-4851-967b-bdcbcf05f0af' requested u'registration_request'
2016-08-24 09:50:10.892 [IPControllerApp] client::client '0b325707-799e-454b-ad86-835332894597' requested u'registration_request'
2016-08-24 09:50:11.295 [IPControllerApp] client::client '8b888d73-d638-4816-b87b-2836bcbed5dc' requested u'registration_request'
2016-08-24 09:50:11.385 [IPControllerApp] client::client '9f3a1367-602a-4cd0-ac5a-ade86170832c' requested u'registration_request'
2016-08-24 09:50:11.393 [IPControllerApp] client::client '49f9ce2f-66d4-4727-8554-f97dbff2472c' requested u'registration_request'
2016-08-24 09:50:11.809 [IPControllerApp] registration::finished registering engine 8:d4346b48-c2df-42c0-b35c-afee186a4d24
2016-08-24 09:50:11.810 [IPControllerApp] engine::Engine Connected: 8
2016-08-24 09:50:11.827 [IPControllerApp] registration::finished registering engine 4:e843f060-c459-44d2-86b3-5e9c1ac59ec0
2016-08-24 09:50:11.828 [IPControllerApp] engine::Engine Connected: 4
2016-08-24 09:50:11.844 [IPControllerApp] registration::finished registering engine 6:db91ec59-1d2e-4118-97e0-ecebd1c91502
2016-08-24 09:50:11.845 [IPControllerApp] engine::Engine Connected: 6
2016-08-24 09:50:11.853 [IPControllerApp] registration::finished registering engine 3:b1af984a-2fa3-4e3e-a147-bd4ad1ac7da3
2016-08-24 09:50:11.854 [IPControllerApp] engine::Engine Connected: 3
2016-08-24 09:50:11.868 [IPControllerApp] registration::finished registering engine 5:843ab55a-bd6c-495a-82c1-772c12c58cf0
2016-08-24 09:50:11.869 [IPControllerApp] engine::Engine Connected: 5
2016-08-24 09:50:11.885 [IPControllerApp] registration::finished registering engine 7:d0dd2841-1084-4caf-9d6c-6b3f9a1d214e
2016-08-24 09:50:11.889 [IPControllerApp] engine::Engine Connected: 7
2016-08-24 09:50:11.970 [IPControllerApp] client::client '28ed44c1-edf6-47f6-9f54-ef8f85d54d42' requested u'registration_request'
2016-08-24 09:50:12.082 [IPControllerApp] client::client 'f5fe9271-c53a-4f87-8931-6c615140ff29' requested u'registration_request'
2016-08-24 09:50:12.267 [IPControllerApp] client::client 'e48ec3a7-2252-49ba-9fa0-125dd9511dc7' requested u'registration_request'
2016-08-24 09:50:12.288 [IPControllerApp] client::client '7c82bba7-a05c-4298-9220-58404eb27b00' requested u'registration_request'
2016-08-24 09:50:12.704 [IPControllerApp] client::client '024939aa-51f5-44ae-af35-cdd03c522b50' requested u'registration_request'
2016-08-24 09:50:12.709 [IPControllerApp] client::client '3bc8cf8a-18d7-43a8-8f5d-1e85bea90b75' requested u'registration_request'
2016-08-24 09:50:12.860 [IPControllerApp] client::client 'd15054ab-ce1e-41e3-95f6-36385f6a60ed' requested u'registration_request'
2016-08-24 09:50:13.000 [IPControllerApp] client::client 'f546cf58-5c1f-40dd-93df-bee32ffd5365' requested u'registration_request'
2016-08-24 09:50:13.185 [IPControllerApp] client::client '5d2cbfac-74fa-4253-97bb-b0086c593da6' requested u'registration_request'
part 3
2016-08-24 09:50:14.809 [IPControllerApp] registration::finished registering engine 13:4745edf2-df44-4971-b812-5755d285b1f7
2016-08-24 09:50:14.810 [IPControllerApp] engine::Engine Connected: 13
2016-08-24 09:50:14.820 [IPControllerApp] registration::finished registering engine 16:8b888d73-d638-4816-b87b-2836bcbed5dc
2016-08-24 09:50:14.827 [IPControllerApp] engine::Engine Connected: 16
2016-08-24 09:50:14.837 [IPControllerApp] registration::finished registering engine 12:3a0a81a1-a499-4c12-8166-c2406af92398
2016-08-24 09:50:14.837 [IPControllerApp] engine::Engine Connected: 12
2016-08-24 09:50:14.842 [IPControllerApp] registration::finished registering engine 15:0b325707-799e-454b-ad86-835332894597
2016-08-24 09:50:14.843 [IPControllerApp] engine::Engine Connected: 15
2016-08-24 09:50:14.847 [IPControllerApp] registration::finished registering engine 17:9f3a1367-602a-4cd0-ac5a-ade86170832c
2016-08-24 09:50:14.848 [IPControllerApp] engine::Engine Connected: 17
2016-08-24 09:50:14.853 [IPControllerApp] registration::finished registering engine 9:980fea61-7d2c-4710-abbf-5640da24f9e2
2016-08-24 09:50:14.855 [IPControllerApp] engine::Engine Connected: 9
2016-08-24 09:50:14.866 [IPControllerApp] registration::finished registering engine 10:9810f9ea-fcac-485f-923d-70200142c0d5
2016-08-24 09:50:14.869 [IPControllerApp] engine::Engine Connected: 10
2016-08-24 09:50:14.880 [IPControllerApp] registration::finished registering engine 11:3764babd-e59b-48ba-b6ac-73106d9ac663
2016-08-24 09:50:14.881 [IPControllerApp] engine::Engine Connected: 11
2016-08-24 09:50:14.885 [IPControllerApp] registration::finished registering engine 18:49f9ce2f-66d4-4727-8554-f97dbff2472c
2016-08-24 09:50:14.885 [IPControllerApp] engine::Engine Connected: 18
2016-08-24 09:50:14.898 [IPControllerApp] registration::finished registering engine 14:ad652bf1-cc72-4851-967b-bdcbcf05f0af
2016-08-24 09:50:14.898 [IPControllerApp] engine::Engine Connected: 14
2016-08-24 09:50:14.998 [IPControllerApp] client::client 'f73b9932-7833-41f7-86af-af747f6a4ec9' requested u'registration_request'
2016-08-24 09:50:15.029 [IPControllerApp] client::client '810f6b3f-3c42-49f2-bc48-da71ac0a6f0a' requested u'registration_request'
2016-08-24 09:50:15.048 [IPControllerApp] client::client '280b2659-58e4-40b2-8154-bfe1a6906c14' requested u'registration_request'
2016-08-24 09:50:15.063 [IPControllerApp] client::client '68ca5004-d533-4886-a91a-e5503b24492a' requested u'registration_request'
2016-08-24 09:50:15.168 [IPControllerApp] client::client '112174a1-46ef-43e9-af50-b52a16c8761c' requested u'registration_request'
2016-08-24 09:50:15.170 [IPControllerApp] client::client '86d675cc-2232-4b70-994b-6b72bf98237b' requested u'registration_request'
2016-08-24 09:50:15.299 [IPControllerApp] client::client '7230053c-da66-4b02-99e4-54e8217be517' requested u'registration_request'
2016-08-24 09:50:15.301 [IPControllerApp] client::client '909eceef-eece-4514-83c4-b3784bcbfe1d' requested u'registration_request'
2016-08-24 09:50:17.490 [IPControllerApp] CRITICAL | Received signal 2, shutting down
2016-08-24 09:50:17.490 [IPControllerApp] CRITICAL | terminating children...
Isaac Overcast
@isaacovercast
Aug 24 2016 17:10
@edgardomortiz_twitter Can we try one more thing:
rm ipyrad_log.txt
ipyrad -p params-test.txt -s 1 -f -d
This will remove the log file, then generate a new one. Please post the contents of ipyrad_log.txt (should be pretty short). Thanks for your help solving this!
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 17:13
2016-08-24 12:13:18,681            pid=17964          [assembly.py]      ERROR      shutdown warning: local variable 'ipyclient' referenced before assignment
I saw that in the log everytime, I should have posted it before
Isaac Overcast
@isaacovercast
Aug 24 2016 17:22
Is that all the content of ipyrad_log.txt? with the -d flag? I would expect there to be much more! Very curious....
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 17:27
sorry, I forgot the -d, here it goes:
2016-08-24 12:13:18,681            pid=17964          [assembly.py]      ERROR      shutdown warning: local variable 'ipyclient' referenced before assignment
2016-08-24 12:24:24,779            pid=21751          [parallel.py]      INFO       ['ipcluster', 'start', '--daemonize', '--cluster-id=ipyrad-21751', '--engines=Local', '--profile=default', '--n=48']
2016-08-24 12:25:03,772            pid=21751          [assembly.py]      INFO       IPyradWarningExit:     No ipcluster instance found. This may be a problem with your installation
    setup. I would recommend that you contact the ipyrad developers.

2016-08-24 12:25:03,775            pid=21751          [assembly.py]      ERROR      shutdown warning: local variable 'ipyclient' referenced before assignment
2016-08-24 12:25:03,775            pid=21751          [parallel.py]      INFO       Shutting down [ipyrad-21751] remote Engines
Isaac Overcast
@isaacovercast
Aug 24 2016 17:34
Thanks, that looks more like what i expected, still doesn't get very far. The shutdown warning is a cascade effect from the real problem which is the cluster isn't starting, nothing to worry about.
Deren Eaton
@dereneaton
Aug 24 2016 17:40
oh, the ipcontroller log file is super weird. It is using a complicated setup that is definitely not "default". I wonder why.
oh wait, no I was looking at the wrong file. lol. It looks pretty much like mine.
Isaac Overcast
@isaacovercast
Aug 24 2016 17:43
One thing I can think is that internally inside get_client() we are catching IOError and then raising the IPyradWarningExit(). There could be something happening to raise the IOError that we're obscuring or not seeing.
Deren Eaton
@dereneaton
Aug 24 2016 17:44
I thought of that too. IOError is a little vague, maybe we're catching something else. I could push a quick version that explicitly prints everything that's going. Still doesn't explain why it worked in the API though.
@edgardomortiz_twitter Try updating through conda again and running it. It should print more information about the error now.
now on : ipyrad-0.3.33-4
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 18:19
ls5 emo347@nid00031: /scratch/02728/emo347/ddRAD/diplo/e_pseudoreference_ipyrad0332 $ ipyrad -p params-ast.txt -s 1 -f -d

  ** Enabling debug mode **

 -------------------------------------------------------------
  ipyrad [v.0.3.33]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  New Assembly: ast
Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-ipyrad-47098-client.json
Connection file '~/.ipython/profile_default/security/ipcontroller-ipyrad-47098-client.json' not found.
You have attempted to connect to an IPython Cluster but no Controller could be found.
Please double-check your configuration and ensure that a cluster is running.

    No ipcluster instance found. This may be a problem with your installation
    setup. I would recommend that you contact the ipyrad developers.
ipyrad_log.txt:
2016-08-24 13:15:55,614            pid=47098          [parallel.py]      INFO       ['ipcluster', 'start', '--daemonize', '--cluster-id=ipyrad-47098', '--engines=Local', '--profile=default', '--n=48']
2016-08-24 13:16:34,684            pid=47098          [assembly.py]      INFO       IPyradWarningExit:     No ipcluster instance found. This may be a problem with your installation
    setup. I would recommend that you contact the ipyrad developers.

2016-08-24 13:16:34,687            pid=47098          [assembly.py]      ERROR      shutdown warning: local variable 'ipyclient' referenced before assignment
2016-08-24 13:16:34,688            pid=47098          [parallel.py]      INFO       Shutting down [ipyrad-47098] remote Engines
Deren Eaton
@dereneaton
Aug 24 2016 18:20
damn, that didn't really help things.
Isaac Overcast
@isaacovercast
Aug 24 2016 18:21
What about just trying this:
ipcluster start --daemonize --cluster-id=ipyrad-12345 --engines=Local --profile=default --n=48
It's something about cluster starting is getting fritzed up.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 18:26
I ran that command before running the ipyrad command again, same result
Deren Eaton
@dereneaton
Aug 24 2016 18:31
The problem might be that the "timeout" call to ipp.Client() doesn't work. It seems to take the same amount of time to raise an error whether is set timeout=1 or timeout=100
Isaac Overcast
@isaacovercast
Aug 24 2016 18:31
That's so bizarre....
Deren Eaton
@dereneaton
Aug 24 2016 18:31
Actually, it looks like they've been working on it: ipython/ipyparallel#185
Deren Eaton
@dereneaton
Aug 24 2016 18:38
ipyparallel 5.2 is up on conda, so you could try updating, but the fix looks like its more recent.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 18:39
I reinstalled conda this morning, everything was running with ipyparallel 5.2
Deren Eaton
@dereneaton
Aug 24 2016 18:52
well, I think we found the problem. I think I can make a work around that might be useful in other situations as well. I'll make a CLI arguments that allows you to run connect to an existing ipcluster instance.
Edgardo M. Ortiz
@edgardomortiz_twitter
Aug 24 2016 19:03
Great, I am glad this helped
Deren Eaton
@dereneaton
Aug 24 2016 19:09
OK, here's how the new option works (though it may change in the future). You have to create the ipcluster instance on your own, which means that if you're connecting to multiple nodes you'll have to add the --engines=MPI and --ip=* arguments when you start it. The important thing, however, is that the you have to set --profile=ipyrad. This just names this instance to be different from the default. This way you could modify the ipyrad profile if you wanted to set some fancy default arguments. Now ipyrad has an argument --ipcluster, which when turned on will tell it to skip trying to create an ipcluster instance and to instead connect to the existing ipcluster instance with profile=ipryad. Its actually really nice to run it this way if users don't find ipcluster to be too scary, since the startup is much faster. We may need to add some cleanup in the code to make sure failed jobs abort etc., but that can be done later.
## start an ipcluster instance with --profile=ipyrad
$ ipcluster start --n 48 --profile=ipyrad 

## run ipyrad with --ipcluster flag so it knows to look for that specific ipcluster instance
$ ipyrad -p params-test.txt -s 2 --ipcluster
a few more tests then I'll push the update.
Deren Eaton
@dereneaton
Aug 24 2016 19:15
of course, it's up to you to remember to run ipcluster stop --profile=ipyrad when you're done. And you should add the --daemonize flag to your ipcluster start command to make it run in the background. Once you get the hang of it, though, ipcluster is really pretty nice.
v.0.3.34 is now up, with a new option to work around ipyparallel timeout bug until it is fixed.
Deren Eaton
@dereneaton
Aug 24 2016 19:22
@edgardomortiz_twitter If you are submitting a job to the cluster and using this method I would stick in a sleep command to make sure the ipcluster instance that is sent to the background is started before ipyrad tries to connect to it, since the delay in it starting up seems to the source of the error message. e.g.,
ipcluster start --n 48 --profile=ipyrad --daemonize
sleep 60
ipyrad -p params-test.txt -s 1 --ipcluster
Zac Forsman
@zacforsman
Aug 24 2016 20:34
Hi folks, what is the easiest way to exclude or add samples other than moving files to a new directory and starting from scratch? -branching?
Isaac Overcast
@isaacovercast
Aug 24 2016 22:21
Hi zac, yes branching is the best way to do this. Did you find the documentation for branching on the readthedocs site? Let me know if you have any questions about it.