These are chat archives for openworm/ChannelWorm

28th
Jul 2015
Travis Jacobs
@travs
Jul 28 2015 15:41
@/all
This is going to be kind of a wall of text, so I'll try to keep it organized.
There has been some work recently to begin a generic (i.e. not model validation) testing suite in ChannelWorm.
@gsarma has created a milestone for this purpose, which will be useful to organize related issues going forward.

Framework

I have looked into the testing frameworks a little more, and PyTest (which has recently been adopted in PyOpenWorm) can actually run tests written with any other framework (i.e. Nose, UnitTest), and is the most extensible (which @kevcmk can attest to, I believe), so I think it makes sense to start with PyTest in this project as well.
Travis Jacobs
@travs
Jul 28 2015 15:48

Data validation tests

@VahidGh & @miladjafary: any data validation tests (ex: #131) will require the presence of the database. Does this interfere with the way we want to sync the live app and the github repo? How do you think we could do this?

Test organization

The django app seems to need its tests in a particular location on disk, so I'm not sure how to handle this off the top of my head. I have a feeling there is going to be a PyTest plugin that can help us with this; it would be nice to have everything in the tests/ directory if we could.
Stephen Larson
@slarson
Jul 28 2015 15:53
Hi @/all -- just want to sneak this in here -- we've moved the ChannelWorm repo under the OpenWorm organization! http://github.com/openworm/ChannelWorm -- thanks to @VahidGh for doing this. Nothing should change organizationally or functionally with the project, but there will probably be some URL changes we'll need to take note of. Cheers!
Travis Jacobs
@travs
Jul 28 2015 15:55
@slarson Awesome!
Thanks @VahidGh; you rock :+1:
Travis Jacobs
@travs
Jul 28 2015 16:03
Alright, so my braindump above is done. I'll raise some issues to go along with it, but I wanted to see if I could kick off some discussion on those points :)
Chee Wai Lee
@cheelee
Jul 28 2015 16:11
@VahidGh Cool stuff! Thanks!
@travs I think it can be made to be more flexible than that. I'm still working my way through the django tutorial using PythonAnywhere but once I get a handle on things, I might be of some help in this department. My idea is to consider starting a new common sub-project to host the framework for testing OpenWorm components (python initially) as well as a performance-regression framework.
Vahid Ghayoomie
@VahidGh
Jul 28 2015 16:15
@slarson, you're welcome. I hope this sub-project would be of help to the OpenWorm community :D
@travs, It mainly depends on what data goes where, if data is going to be kept in the PyOW, then we can have tests as same as those for PyOW.
Other functional tests could be done through the django approach.
Travis Jacobs
@travs
Jul 28 2015 16:30
@cheelee Awesome! :D
Regarding your idea for a testing framework subproject, if you're into podcasts check this one out. An author of PyTest talks about how the lib can be used to test non-python code as well. Well worth a listen imo.
@VahidGh Great point :-)
There will certainly be some data in CW that is not exported to PyOW (i.e. at minimum, user data), so there should probably be tests on this.
I'm not up to speed on the db sync process of CW right now. Is the db stored in the github repo at all? Or is it just on the openshift server?
Vahid Ghayoomie
@VahidGh
Jul 28 2015 16:37
I'm trying to keep them synced. It is now manually, but when #134 get done, it would be much easier to do the synchronization when needed.
Now I'm getting daily backups from the server, and pushing to the master if there is any change.
Travis Jacobs
@travs
Jul 28 2015 17:03
@VahidGh Ok gotcha. The reason I ask is because this was the way things were done in PyOpenWorm initially (hosting the binary db on the github repo as well), but it lead to giving PyOpenWorm a very large download size due to the way git keeps track of binary objects (recall your issue: openworm/PyOpenWorm#99).

This repo size becomes a barrier/hidrance to developing locally, and doing a history rewrite can be slightly painful as well.

The solution in PyOpenWorm involves building the database "from source", but since CW's data is manually entered, there are no csv or other source files that can dynamically generate the db, so this solution is not possible in CW.

Travis Jacobs
@travs
Jul 28 2015 17:09
@VahidGh
Before we discuss other potential solutions, do you agree that storing the db in git this way may lead to problems in development in the future?
Chee Wai Lee
@cheelee
Jul 28 2015 17:10
@travs awesome, will check out that podcast. Thanks!
Shreejoy Tripathy
@stripathy
Jul 28 2015 17:10
@travs @VahidGh just to chime in here - neuroelectro has a similar problem with DB and size
we share around a DB dumpfile that needs to be downloaded everytime someone wants to develop locally - but this is a hinderance and isn't as simple as "pip install neuroelectro"
Travis Jacobs
@travs
Jul 28 2015 17:23
@stripathy Ok, that approach makes sense to me though: separating the db from app version control.
What if the dumpfile was automatically downloaded on install? It's still a hindrance due to the size of the download probably?
Vahid Ghayoomie
@VahidGh
Jul 28 2015 17:29
@travs, @stripathy, currently the db is ~300 KB, I don't think it's going to be more than a MB, as data we deal with "in DB" is not really big in size.
So we can handle this using dumpfiles, etc.
The main problem we have here is the size of the media directory, where the images are kept in, and have much larger size (also it's the same for 'static' directories).
Shreejoy Tripathy
@stripathy
Jul 28 2015 18:22
@VahidGh - for neuroelectro we have a similar problem in that we have to store GBs worth of article full texts - what we do is store them in an external directory and then use a django symlink field to link out to these
my sense is that the channelworm DB will be relatively small for the coming months, no?
Vahid Ghayoomie
@VahidGh
Jul 28 2015 18:54
@stripathy, yeah, I guess the db is not going to be more than 1-2 MB
But the external link for images is a good idea.
I'm thinking about how we can handle this when uploading a file!