Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 18 20:22
    svdavid closed #253
  • Aug 18 20:22
    svdavid opened #253
  • Jul 16 19:50
    jacobpennington closed #252
  • Jul 16 19:50
    jacobpennington synchronize #252
  • Jul 16 19:49
    jacobpennington opened #252
  • Jun 17 22:01
    svdavid closed #251
  • Jun 17 22:01
    svdavid opened #251
  • May 21 03:24
    svdavid closed #250
  • May 21 03:24
    svdavid opened #250
  • Apr 10 00:42
    svdavid closed #249
  • Apr 10 00:04
    jacobpennington opened #249
  • Apr 10 00:04
    jacobpennington review_requested #249
  • Apr 09 23:52
    jacobpennington closed #248
  • Apr 09 23:50
    jacobpennington opened #248
  • Jan 29 20:00
    jacobpennington closed #247
  • Jan 25 00:25
    jacobpennington review_requested #247
  • Jan 25 00:25
    jacobpennington opened #247
  • Jan 23 18:45
    svdavid closed #246
  • Jan 23 18:09
    crheller commented #246
  • Jan 23 02:41
    jacobpennington edited #246
Stephen D
@svdavid
Hi everyone! I'm hoping this will be a good place to move NEMS-related discussions. It looks like we can integrate it to track activity in the repo, which may be useful and/or could be a pain.
Brad Buran
@bburan
I'm here!
Stephen D
@svdavid
Hurray! Another live person!
Brad Buran
@bburan
I haven't done any work on NEMS lately, so haven't really had anything to say.
Stephen D
@svdavid
Don't worry, I think we're all guilty parties. Anyway, this is the "support" channel.
Brad Buran
@bburan
But, @crheller and I had a discussion an hour or so ago regarding the need to revamp the Signal, Recording and Epoch systems.
Stephen D
@svdavid
ANd nems-dev is the developer channel.
And right, I was following up on exactly that topic.
jacobpennington
@jacobpennington
Sorry, I'm alive as well. Guess I forgot to set up notifications so I didn't notice anyone was using this.
Stephen D
@svdavid
Ha yes, notifications are a bit of a hassle. The annoying thing is that if you have gitter open in a browser then it won't send notifications to the app on your phone. I don't see a way for it to not do that.
Daniela Saderi
@dasaderi
Use Rambox for all the things in one bucket.
Brad Buran
@bburan
Interesting.
I had not heard of that before. I just "pin" some tabs in Firefox so they open every time I load my browser.
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo
Is there a reason why TiledSignal does not have a ._modified_copy( ) method? seems trivial to implement. Also I tough that signals were inmutable, playing with them recently, I realized you can overide any field of the class, is this a recent modification??
jacobpennington
@jacobpennington
They were originally intended to be immutable but I think that constraint was lifted at some point for one reason or another. I don't recall exactly. Not sure why TiledSignal is missing that method but should be fine to add.
Stephen D
@svdavid
@Mateo-Lopez-Espejo yes, please feel free to add the method. Probably no one has needed it yet.
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo

I am having some issues trying to fit a slightly more complex model:
'wc.2x2.c-stp.2-fir.2x15-lvl.1-stategain.S-dexp.1'
I am getting the following error

Traceback (most recent call last):
  File "/auto/users/mateo/miniconda3/envs/mynems/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/mateo/code/context_probe_analysis/single_cpp_processing.py", line 28, in single_cpp_processing
    ctx, log_xf = xforms.evaluate(xfspec)
  File "/auto/users/mateo/NEMS/nems/xforms.py", line 132, in evaluate
    context = evaluate_step(xfa, context)
  File "/auto/users/mateo/NEMS/nems/xforms.py", line 89, in evaluate_step
    new_context = fn(**args)
  File "/auto/users/mateo/NEMS/nems/xforms.py", line 530, in fit_basic_init
    tolerance=tolerance, max_iter=700, norm_fir=norm_fir)]
  File "/auto/users/mateo/NEMS/nems/initializers.py", line 162, in prefit_LN
    modelspec = init_dexp(est, modelspec)
  File "/auto/users/mateo/NEMS/nems/initializers.py", line 372, in init_dexp
    rec = ms.evaluate(rec, fit_portion)
  File "/auto/users/mateo/NEMS/nems/modelspec.py", line 203, in evaluate
    kwargs = {**fn_kwargs, **m['phi']}  # Merges both dicts
KeyError: 'phi'

so what I understand is happening is that during the prefiting. The program is looking for the phy of a set of specific modules, which do not include stategain. Is this about the order of the modules? is there a reason why stategain has to be last? I am guessing that the order of the modules might generate a difference in this case, but that is a gut feeling.

Stephen D
@svdavid
Yes, probably has to do with the order of modules. You can tweak init_dexp to set any missing phi entries to the mean of the prior. Have you seen that code? There are several other places where this happens. Eg in fit_basic, this is how phi gets initialized
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo
is there any reason for not generating predictions from the original recording and instead doing it only with the estimation and validation recordings?
Stephen D
@svdavid
Well, interpreting predictions in the two sets is different. est predictions can be over-fit, while val can't. If you do n-fold cross-validation, then you do effectively generate a validation set that matches the size of rec.
And you can certainly generate predictions for rec, but you have to be careful with interpretation.
(replying to @Mateo-Lopez-Espejo)
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo

So I am trying to push a bunch of jobs to the cluster and I am getting this error

QUEUEID=558037 QPID=22094 TMP=/tmp/558037
about to get codehash from mysql
about to check for codehash stuff
about to check if default should be run
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/nems/bin
MATLABPATH: 
COMMAND: nice -n19 /auto/users/mateo/miniconda3/envs/nemsenv2/bin/python /auto/users/mateo/context_probe_analysis/cluster_script.py BRT37b-39-1 310 wc.2x2.c-fir.2x15-lvl.1-dexp.1
[nems.configs.defaults INFO] Saving log messages to /tmp/nems/NEMS 2018-11-13 162008.log
[nems_db INFO] No value specified for: NEMS_DB_API_HOST. Using default value in nems_db.configs.defaults
[nems_db INFO] No value specified for: NEMS_DB_API_PORT. Using default value in nems_db.configs.defaults
[nems_db INFO] No value specified for: NEMS_RESULTS_DIR. Using default value in nems_db.configs.defaults
[nems_db INFO] No value specified for: DEFAULT_EXEC_PATH. Using default value in nems_db.configs.defaults
[nems_db INFO] No value specified for: DEFAULT_SCRIPT_PATH. Using default value in nems_db.configs.defaults
[nems.xforms INFO] Evaluating: nems_db.xform_wrappers.baphy_load_wrapper
[nems_db.xform_wrappers INFO] loader=env.fs100.cst
Starting QUEUEID=558037
Running single_oddball_processing with parameters (BRT37b-39-1,310,wc.2x2.c-fir.2x15-lvl.1-dexp.1)
Traceback (most recent call last):
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 727, in _read_query_result
    result.read()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 1066, in read
    first_packet = self.connection._read_packet()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 683, in _read_packet
    packet.check_error()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1")

The above exception was the direct cause of the following exception:

continues

Traceback (most recent call last):
  File "/auto/users/mateo/context_probe_analysis/cluster_script.py", line 54, in <module>
    ctx = csp.single_cpp_processing(cellid, batch, modelspec_name)
  File "/auto/users/mateo/context_probe_analysis/single_cpp_processing.py", line 28, in single_cpp_processing
    ctx, log_xf = xforms.evaluate(xfspec)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/NEMS/nems/xforms.py", line 132, in evaluate
    context = evaluate_step(xfa, context)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/NEMS/nems/xforms.py", line 89, in evaluate_step
    new_context = fn(**args)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/xform_wrappers.py", line 197, in baphy_load_wrapper
    loadkey=loadkey, siteid=None, **options)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/xform_wrappers.py", line 180, in generate_recording_uri
    recording_uri = nb.baphy_load_recording_uri(**options)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/baphy.py", line 1375, in baphy_load_recording_uri
    options = fill_default_options(options)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/baphy.py", line 969, in fill_default_options
    rawid=rawid)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/db.py", line 1202, in get_stable_batch_cells
    cellids, rawid = get_stable_batch_cells(batch, siteid, rawid)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/db.py", line 1173, in get_stable_batch_cells
    d = pd.read_sql(sql=sql, con=engine, params=params)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pandas/io/sql.py", line 397, in read_sql
    chunksize=chunksize)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pandas/io/sql.py", line 1063, in read_query
    result = self.execute(*args)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pandas/io/sql.py", line 954, in execute
    return self.connectable.execute(*args, **kwargs)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2075, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
continues

  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 727, in _read_query_result
    result.read()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 1066, in read
    first_packet = self.connection._read_packet()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/connections.py", line 683, in _read_packet
    packet.check_error()
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/auto/users/mateo/miniconda3/envs/nemsenv2/lib/python3.6/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1") [SQL: 'SELECT cellid FROM NarfData WHERE 1 AND batch=%s AND label = %s AND cellid like %s AND rawid IN %s'] [parameters: ('310', 'parm', 'BRT37b%', ())] (Background on this error at: http://sqlalche.me/e/f405)
I have checked that all the paths are right, both the script that I am running, whichs works perfectly locally, as well as the environment used for it
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo
I just realized what is the issue... some faulty config file for nems_db. Is there some copypasta config file that I can use?
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo
solved. But lead to an ulterior question. Right now there are two apparent locations for config files within nems_db, at nems_db/configs/ and at nems_db/nems_db/configs/ what is the best practice with this two locations?
jacobpennington
@jacobpennington
There should only be one location for nems_db-specific settings, nems_db/nems_db/configs, and a separate one for nems_db/nems_web/configs. Master branch on github should reflect that if you go through the file browser online. There was previously a configs directory in the top-level nems_db but it was removed a while ago, so it probably just didn't get deleted in your local directory. Should be safe to delete it.
Stephen D
@svdavid

The issue seems to be happening at this line:

File "/auto/users/mateo/miniconda3/envs/nemsenv2/nems_db/nems_db/db.py", line 1173, in get_stable_batch_cells
    d = pd.read_sql(sql=sql, con=engine, params=params)

Simplest thing is to have python print out "sql" immediately before executing to see what's wrong with the query syntax.

@crheller wrote that function and may be able to help. Probably something to do with identifying the right rawids in celldb
Brad Buran
@bburan
@Mateo-Lopez-Espejo If you are working off of a repo you cloned a while back, it does not always prune files that were permanently relocated in the repo. So, if configs used to live at nems_db/nems_db/configs then was moved to nems_db/configs by someone else's commit, it may not be reflected even when you update from master.
You'd want to do a git clean -fd (but be very very careful when doing so as it removes all untracked files and directories).
Think of it as a rm -rf *
Mateo-Lopez-Espejo
@Mateo-Lopez-Espejo
ok got it, thank you.
Brad Buran
@bburan
Sometimes I get confused about whether a file should be there or not, so I will go to the Github page for NEMS and browse the code to see if the file/folder is there. I run into this issue all the time (with other repositories) since I like to rearrange my code a lot.
Brad Buran
@bburan
A very useful site for creating new Github repos (it creates a custom gitignore file for you). https://www.gitignore.io/
PhantomSpike
@PhantomSpike
Hi guys,
Nice to meet you virtually! I am Alex, PhD student in the lab of Andy King at Oxford. I was just chatting with Stephen who is visiting our lab and we were discussing trying out NEMS with our data. I will be be annoying you with questions over here haha.
jacobpennington
@jacobpennington
Cool, welcome!
Stephen D
@svdavid
@PhantomSpike If you haven't found it yet, a good place to start is nems/scripts/demo_script.py
PhantomSpike
@PhantomSpike

Hey!

I managed to install it successfully. I had some problems but I think it was just because anaconda was owned by root and not me as a user. Anyway, now it is working.

What would be the best way to get started? Going through the demo_script.py and figuring which part does what, changing parameters and just playing with it? I should also probably read the table of contents and understand how to format the data, what is the expected input/output and so on.

Is there a recommended data set that I can start with? I have some of my own but if you guys have something that you know works well I wouldn't mind just going for that in the beginning.

I will stick with local changes only until I figure out how things work but when I get into it I might start adding some commits later on.

Thanks for developing NEMS, it looks quite useful and fun :)

jacobpennington
@jacobpennington
You can get some samples from our dataset by running nems/scripts/download_demo_data.py. It will download them to nems/recordings/ by default, and they're already set up to be loaded using nems.recording.load_recording().
PhantomSpike
@PhantomSpike
Great, thanks
Stephen D
@svdavid
I know there hasn't been much talk going on here recently. But just to make sure it's out there somewhere, we've merged many changes into the master branch from dev. And for the near future, we'll keep "master" as the active branch, since no one has been using it as a "stable" branch.
Brad Buran
@bburan
We also have documentation up at https://nems.readthedocs.io
Hopefully we can continue fleshing out the API documentation (https://nems.readthedocs.io/en/latest/api_ref/index.html) since that's a big gap in the documentation. We are using Sphinx autodoc directives, but it has to be manually set up for each module (once set up, it will automatically find all classes and functions and pull in the docstrings).
See https://nems.readthedocs.io/en/latest/api_ref/signal.html#module-nems.signal for an example of autodoc for the nems.signal module.
This is the rst file that's needed to trigger the documentation for that module: https://raw.githubusercontent.com/LBHB/NEMS/master/docs/source/api_ref/signal.rst
8 lines including whitespace.