These are chat archives for ROBelgium/MSNoise

Apr 2017
Tobias Megies
Apr 27 2017 13:03
we're trying to scan some folder
2017-04-27 15:04:42 [DEBUG] ERROR: (pymysql.err.InternalError) (1176, u"Key
'da_index' doesn't exist in table 'data_availability'") [SQL: u'SELECT
data_availability.ref AS data_availability_ref, AS
data_availability_net, data_availability.sta AS data_availability_sta,
data_availability.comp AS data_availability_comp, data_availability.path AS
data_availability_path, data_availability.file AS data_availability_file,
data_availability.starttime AS data_availability_starttime,
data_availability.endtime AS data_availability_endtime,
data_availability.data_duration AS data_availability_data_duration,
data_availability.gaps_duration AS data_availability_gaps_duration,
data_availability.samplerate AS data_availability_samplerate,
data_availability.flag AS data_availability_flag \nFROM data_availability USE 
INDEX (da_index) \nWHERE data_availability.path = %(path_1)s AND 
data_availability.file = %(file_1)s AND = %(net_1)s AND 
data_availability.sta = %(sta_1)s AND data_availability.comp = %(comp_1)s \n
LIMIT %(param_1)s'] [parameters: {u'net_1': u'BW', u'param_1': 1, u'file_1':
u'BW.RMOA..EHZ.D.2013.033', u'path_1':
u'/bay200/mseed_online/archive/2013/BW/RMOA/EHZ.D/', u'sta_1': u'RMOA',
u'comp_1': u'EHZ'}]
Any idea?
Thomas Lecocq
Apr 27 2017 13:06
msnoise upgrade_db
for some reason it doens't create the index on install
Tobias Megies
Apr 27 2017 13:07
ah, thanks!
Tobias Megies
Apr 27 2017 13:14
so.. maybe "msnoise install" should automatically also call "msnoise upgrade_db" ;-P
tha naming of init in scan archive is pretty weird i think.. not very intuitive
maybe it should be renamed to something like all?
or any_timestamp
Tobias Megies
Apr 27 2017 13:39
we just came across this error during computing CC

Process Process-1:
Traceback (most recent call last):
  File "/home/tremor/anaconda2/envs/msnoise01/lib/python2.7/multiprocessing/", line 258, in _bootstrap
  File "/home/tremor/anaconda2/envs/msnoise01/lib/python2.7/multiprocessing/", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tremor/git/MSNoise_msnoise01/msnoise/", line 444, in main
    export_allcorr(db, ccfid, allcorr[ccfid])
  File "/home/tremor/git/MSNoise_msnoise01/msnoise/", line 887, in export_allcorr
    df.to_hdf(os.path.join(path, date+'.h5'), 'data')
  File "/home/tremor/anaconda2/envs/msnoise01/lib/python2.7/site-packages/pandas/core/", line 1138, in to_hdf
    return pytables.to_hdf(path_or_buf, key, self, **kwargs)
  File "/home/tremor/anaconda2/envs/msnoise01/lib/python2.7/site-packages/pandas/io/", line 269, in to_hdf
    complib=complib) as store:
  File "/home/tremor/anaconda2/envs/msnoise01/lib/python2.7/site-packages/pandas/io/", line 433, in __init__
ImportError: HDFStore requires PyTables, "No module named tables" problem importing
installing pytables fixed it
maybe a problem in the pytables vendored by pandas.. looks like??
would be nice to have an option to import a config file
could be useful when setting up projects while playing around with the parameters
not having to input config items via web interface
Thomas Lecocq
Apr 27 2017 13:45
pytables it is yes
config --dump and --import is very desired :)
otherwise command line --set param=value is ok for small bit changes
--init is really an init... it's "initialize db with anything there"...
Tobias Megies
Apr 27 2017 13:48
ah.. good to know you can set individual keys from CL
that's good enough i guess, can be bundled in a custom setup shell script
Thomas Lecocq
Apr 27 2017 13:50
yeah, i actually kinda liked the way salvus guyz showed one of the config stuff, you call some func in the code and it creates a dummy config file with all params + explanations... I guess parsed by configobj after... it could be the way to dump and load config
actually could be useful when willing to dump local sqlite and import in HPC mysql ... but then it's more a sql-dump
this said, scanning & defining jobs is really fast now (as fast as disk IO)

ah.. good to know you can set individual keys from CL

I think it's in the documentation :)

Tobias Megies
Apr 27 2017 14:04
So, about the pytables.. should it be installed for msnoise? Because in that case, it's missing in the install instructions
Thomas Lecocq
Apr 27 2017 14:07
yes, I thought it was automatically part of pandas :)
Tobias Megies
Apr 27 2017 14:08
well.. looks like it is but seems to be broken??
i also don't quite get it