These are chat archives for connectordb/connectordb

16th
Oct 2016
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:18
@all Thanks so much for this project! I've been dying for exactly this for a huge QS effort I'm starting :D
Daniel Kumor
@dkumor
Oct 16 2016 16:20
That's great! Just so you know, the current release is quite outdated - a new version is very close to release with MANY many improvements
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:21
Good to know! Is there any way I can help ship the next release?
Daniel Kumor
@dkumor
Oct 16 2016 16:21
What OS do you use?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:23
server-side: whatever flavor of linux takes the least amount of effort to config on aws
client-side: windows 10 x64, though I'm considering going back to dual-booting win10/ubuntu
what OS(es) do you need?
Daniel Kumor
@dkumor
Oct 16 2016 16:23
So the new version actually runs fine on windows
meaning that no separate server is needed
I can actually send you the current development version
The only annoyance is that the android app can't really sync to non-server versions yet
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:25
Yes, please, and thank you. Running the server locally doesn't fit my use case, but doing it that way for now will save me money :D
Daniel Kumor
@dkumor
Oct 16 2016 16:25
it will certainly help you evaluate the software before dedicating a server to it!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:25
I'm already using RescueTime for collecting that sort of data anyway.
Daniel Kumor
@dkumor
Oct 16 2016 16:26
gotcha - do you have a specific use case?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:26
evaluate shmevaluate...the only other feasible alternative I've found involves way more coding than I'd like
laptoplogger_install_fixed should be a fairly recent installer
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:27
I'm a recovering info addict (meant not in the "oh I'm such an info junkie" way, but in the "this is a problem that was ruining my life" way)
Daniel Kumor
@dkumor
Oct 16 2016 16:27
so how is gathering more data supposed to help?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:27
RescueTime premium comes with a distraction blocker in the browser that kicks in based on quotas
science is only recently starting to treat addiction as a learning disorder, which opens the door for lots of new ideas on how to treat it
1) I'll be publishing my data for others to analyze for whatever they come up with at informationaddictionrecovery.org (nothing there at the moment)
Daniel Kumor
@dkumor
Oct 16 2016 16:30
aah
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:30
2) I can use the data to better identify antecedents to problem behaviors
3) automate tapping my support system (or my awareness) when the data suggests that maybe I'm in a headspace where I could use a friend
(only really been learning how to be aware of my emotions for a few months, so that'd be really helpful right now)
Daniel Kumor
@dkumor
Oct 16 2016 16:32
got it - you can actually directly set devices to public in ConnectorDB, meaning that anyone can access the data directly - although there are some bugs in this use case that have yet to be fixed
I'm guessing you're planning on using the Python API to perform data analysis?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:33
oh nice. Bugs = ways I can start diving in :)
yeah
Daniel Kumor
@dkumor
Oct 16 2016 16:33
great!
The link I sent you is fairly recent
will allow you to get acquainted with how it works
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:34
4) i'm looking for someone to help me build this to help me improve my emotional intelligence (mostly to recognize when I'm anxious): eqradio.csail.mit.edu
5) I'm also developing a software/product design methodology that incorporates ^^that to take into account the user's emotional state in a more scientifically rigorous way
Daniel Kumor
@dkumor
Oct 16 2016 16:35
Sounds great!
ConnectorDB is always open to contributions and improvements, so please at least file a bug report if you find something annoying/would like to see someting implemented
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:37
Oh...and the front-end for data-entry! EXACTLY what I wasn't finding anywhere
Daniel Kumor
@dkumor
Oct 16 2016 16:37
The manual data entry is turning out to be one of the most useful parts in my experience using CDB
Especially star ratings - rating several dimensions of your life daily allows you to correlate these with your data
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:39
Yep. That's one aspect of it. I'll also be storing the data a behaviorist I'm working with generates, along with to-do lists (and what gets completed when), and daily gratitude lists.
Daniel Kumor
@dkumor
Oct 16 2016 16:39
Be aware of one of the weaknesses in ConnectorDB: streams are append only
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:40
Can you give an example of a use case where that could get in the way?
Daniel Kumor
@dkumor
Oct 16 2016 16:40
Sure! For example using a stream as a way to store your schedule
you can't go back and modify your schedule
streams are built for sensor-style data
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:42
ooooh ok. Is that even a solvable problem or just a misfit between streams and schedule use cases?
Daniel Kumor
@dkumor
Oct 16 2016 16:43
You can always put your whole schedule into one datapoint - a stream would then be the history of your schedules
The underlying reason is that streams can be listened to, and accessed as arrays
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:43
like...is there a system out there with streams that aren't append only and/or plans to modify the CDB streams to be modifiable?
Daniel Kumor
@dkumor
Oct 16 2016 16:43
there are plans
but not in this version yet
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:44
Ok. Can you think of a use case that is maybe a more natural fit for streams where mutability would be nice?
using streams for schedules feels like a mismatch
Daniel Kumor
@dkumor
Oct 16 2016 16:45
certainly
the only annoyance I have found was when inputting ratings - you can't retroactively insert a mood rating for yesterday if you already added one for today
this is a minor issue, which is why it is not high on the list of priorities
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:47
Yeah. I guess when I think "stream" I think continuous/real-time data.
Daniel Kumor
@dkumor
Oct 16 2016 16:48
precisely - it also gives you the guarantee that when subscribed in python, you always have the most recent dataponit
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:48
It makes me wonder if that's the right concept to use with things like mood/schedules.
Daniel Kumor
@dkumor
Oct 16 2016 16:48
schedule definitely not
It works fine for mood
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:50
Especially if used with eqradio to wirelessly detect current emotional state ;)
Daniel Kumor
@dkumor
Oct 16 2016 16:50
yes - that would be a good use case for a stream
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:51
This hit my radar via HN yesterday. May be good inspiration for how to handle the data streams aren't a good fit for. https://github.com/attic-labs/noms
Daniel Kumor
@dkumor
Oct 16 2016 16:53
yes, saw it - it is an interesting concept. Not sure how useful it would be for schedule-type use cases. Currently the plans are to have scheduling/planning be separate from the data-gathering
But before any of the advanced functionality is added, the basic plotting backend needs to be finished. That is the goal for this next release.
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:55
cool. I'm more interested in the dataviz/analysis side of things anyway. I'm tired of wallowing in self-data
Daniel Kumor
@dkumor
Oct 16 2016 16:55
the data is not particularly useful without automated analysis
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:57
sigh don't I know it...been searching for a simple solution to mine the topics in my google/twitter/fb/email histories/logs. all i'm finding is that I don't know quite enough about machine learning to roll my own thing
any support for or plans to support image/video data?
Daniel Kumor
@dkumor
Oct 16 2016 16:58
image data should work fine already - just base64 encode
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:58
ok cool
Daniel Kumor
@dkumor
Oct 16 2016 16:58
I thought about video, but it is a very data-intensive process
Saving fullHD streaming video from a webcam would be tough - I'm sure ConnectorDB could handle it if done cleverly, but I don't recommend it
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 16:59
true. and YouTube's video dataset was crafted using single frames per second cause google determined that was sufficient granularity for ML purposes
Daniel Kumor
@dkumor
Oct 16 2016 17:00
In a sense - it is simply too compute-intensive to do more
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:00
yeah. they found there wasn't much of a payoff for the added computations, either.
I'm going to start buying busted android phones & mounting them around my apt.
As ip cams to stream to CDB
Daniel Kumor
@dkumor
Oct 16 2016 17:01
an image per second?
I'm actually curious how CDB will handle it
It can take thousands of inserts per second
but I never tried it on image data
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:03
cool. I'll let you know how it goes. Also want to do the same with screenshots, too.
Daniel Kumor
@dkumor
Oct 16 2016 17:04
yes, that should be fine. Perhaps an image datatype for streams would be useful. I'll look into it
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:04
could be
Also, are you familiar with this http://github.com/cantino/huginn?
Daniel Kumor
@dkumor
Oct 16 2016 17:05
no, I am not familiar with it - although it seems like a very useful thing to integrate with CDB
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:05
Yeah. Either integration or feature-stealing
Daniel Kumor
@dkumor
Oct 16 2016 17:06
haha - you can already monitor arbitrary data in CDB with the python API - but there is no nice web interface for it
I will actually look into this in more detail, thanks!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:07
CDB seems like a potentially better backend option for them. I just wish someone would port their stuff to python cause there's soooo much ruby bloat in that thing it's hard for a ruby novice like me to really contribute without putting in way more time than I can
Daniel Kumor
@dkumor
Oct 16 2016 17:08
potentially, yes - once this version is released, I'll see if it is possible to integrate the two.
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:08
I can also test running cdb in bash for windows, too.
Daniel Kumor
@dkumor
Oct 16 2016 17:09
The installer I sent you is a full windows executable
no linux dependencies necessary
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:10
right, but it's just the client & not the server?
Daniel Kumor
@dkumor
Oct 16 2016 17:10
nope, it is both
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:10
oh...just saw that
well isn't that special :D
Daniel Kumor
@dkumor
Oct 16 2016 17:10
the client automatically manages the server for you
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:12
suppose I have 2 servers & want to sync a cdb between the two. is there a mechanism for that?
Daniel Kumor
@dkumor
Oct 16 2016 17:12
not yet
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:13
Ok. Know if manually doing it would be relatively easy to pull off?
Daniel Kumor
@dkumor
Oct 16 2016 17:13
it is easy enough to export the data (just get it all in python), so you can just log into both, take the data from one, and insert it into the other
~20 lines of python
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:14
easy enough
Daniel Kumor
@dkumor
Oct 16 2016 17:14
but there is no mechanism for synchronizing two databases in active use - if they both use the same streams
ie, if database 1 nserts into stream 1, 2, 3
database 2 inserts into 4,5,6
then it is fine
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:15
Gotcha. I was really just asking cause I'd still like to test out running the linux server in winbash for the sake of development & wanted an easy way to populate it.
Daniel Kumor
@dkumor
Oct 16 2016 17:15
oh yeah, populating a server is easy
but just so you are aware, the alpha1 is very outdated - it had several issues that are fixed now
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:17
that's what's on GH right now?
Daniel Kumor
@dkumor
Oct 16 2016 17:17
The current release on GH, yes
master is the new version
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:18
Oh ha....I almost never touch actual releases & was confusing "release" with "master"
Daniel Kumor
@dkumor
Oct 16 2016 17:18
CDB is also split across 3 repositories:
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:20
I'm a little confused by the device role descriptions for reader/writer
That's probably because I've read almost none of the cdb docs ;)
Daniel Kumor
@dkumor
Oct 16 2016 17:21
The docs for current version are pretty sparse (working on fixing that)
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:21
Would it be relatively easy for you to open an issue w/a documentation todo list I can start tackling?
Daniel Kumor
@dkumor
Oct 16 2016 17:22
That would be wonderful, thanks!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:22
I think I'm getting hung up on the roles cause I'm not sure of reader device use cases
Daniel Kumor
@dkumor
Oct 16 2016 17:23
reader device can't modify any other devices
it can only read data
writer device can update device properties, as well as reading data
A writer device can also write to downlinks
a user device has all the permissions that a user has - including creation and deletion of devices/streams
A device ALWAYS has full permissions for its own streams - the roles are for data outside of its own streams
so for example, the chrome extension can only access its own data - it can't access your moods, your location history or anything it did not itself write
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:25
yeah
it was confusing for me because I don't have anything in my life right now that operates on my data
but a colored led that reads my moods would be a reader device
Daniel Kumor
@dkumor
Oct 16 2016 17:26
exactly
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:26
what about service integrations?
Daniel Kumor
@dkumor
Oct 16 2016 17:26
for example?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:27
suppose http://gyrosco.pe wanted to support cdb. would a user setup the service as a reader device?
Daniel Kumor
@dkumor
Oct 16 2016 17:28
yes
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:28
aha!
Daniel Kumor
@dkumor
Oct 16 2016 17:28
in fact, the permissions are completely defined in configuration
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:28
Expect PRs from me that are semantic nitpicks XD
Daniel Kumor
@dkumor
Oct 16 2016 17:28
so you are not limited to the default devie roles
sure!
I'd love contributions
these are default permissions
but you can define your own by adding a permissions configuration file
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:30
oh and the invite for ideas was totally unnecessary for me. One of the behaviors I've had to retrain in recovery was compulsively generating/pitching ideas. For at least a few weeks, most convos I started were around a new idea.
Daniel Kumor
@dkumor
Oct 16 2016 17:31
ha - well, PRs are more welcome than ideas, since they don't require me to implement them :)
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:31
hehe of course.
I intend to do both. One thing I'm noticing now that I'm starting to get back into coding after a year off is that anxiety has been my norm since I was 10-12 yrs old & a TON of it comes up when I'm doing something someone else will be seeing. So actually implementing anything will be a therapeutic struggle for me.
Also a chance to actually practice a lot of the dev workflows I've only read about & never actually practiced all that much of (like proper git usage)
Daniel Kumor
@dkumor
Oct 16 2016 17:36
I look forward to PRs! I also hope that the code isn't too much convoluted-spaghetti!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:37
No promises there! One of my biggest barriers to progress right now is premature optimization, which means I'll be overcompensating away from good style.
Daniel Kumor
@dkumor
Oct 16 2016 17:37
I was talking about my code
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:37
But I'll refactor on my own before opening PRs...good python coding practices are another area I'm not well-practiced in
Lol....damn. I was hoping to learn style from your code
Daniel Kumor
@dkumor
Oct 16 2016 17:38
Depends on if you will be constributing to the golang server or the javascript frontend
golang has a well-defined style guideline
and an automated formatter
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:39
oh right...it's a python API but the server's in go!
Daniel Kumor
@dkumor
Oct 16 2016 17:39
yes
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:40
...so maybe I'll start just with helping on the docs while I take a little time to learn go
docs & bugs for now ;)
Daniel Kumor
@dkumor
Oct 16 2016 17:40
sure - any help is welcome
The best way is to use it yourself, and see what the pain points are
You can find a tutorial to the python API here:
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:41
I'll definitely be playing with all of it.
Daniel Kumor
@dkumor
Oct 16 2016 17:42
I'll try to get an early version of the new docs out this week so that they can be contributed to
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:42
the thing I'm most interested in is importing datasets from chat logs, browser/search history, etc. So importing from csv/json
Daniel Kumor
@dkumor
Oct 16 2016 17:42
ah, yes, that was already in the pipeline for a while now
wasy enough to implement in python
but a nice import GUI in the frontend is the ultimate goal
BTW: you can install the chrome extension to gather browsing history
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:44
I think I'll probably avoid the extension for now simply cause my main use case is rescuetime integration & they've got an extension for that
Daniel Kumor
@dkumor
Oct 16 2016 17:45
sounds good to me
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:47
I was already planning on researching UX concepts for data importing this week, but are there any UIs you've got in mind?
I have the same question wrt UI/dataviz frameworks or libs
Daniel Kumor
@dkumor
Oct 16 2016 17:49
The frontend is built with react, and ideally would allow running transforms on the data before importing (my plan was processing all the data in javascript, and inserting using the javascript lib)
similarly, the data visualization uses chartsjs currently
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:51
Time to stop putting off learning React, too XD
Daniel Kumor
@dkumor
Oct 16 2016 17:55
Honestly, right now the biggest issue is the android app - if you think you'd be able to help out with that, it'd be awesome!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 17:55
Any specific issues with it you'd like help with?
Daniel Kumor
@dkumor
Oct 16 2016 17:55
The main issue is allowing sync on wifi
meaning
most people don't want to keep an online server
having a server on your local machine, and an option in android to only sync to your IP when on your home network is needed before this version is released
The android app is also a mess in general
and I don't have enough time to really focus on it
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:00
It's a little hacky, but maybe using http://ngrok.com could provide a quick solution until a wifi-local one's completed
Daniel Kumor
@dkumor
Oct 16 2016 18:01
There is no need for a quick solution - I run ConnectorDB on a internet-facing server
and so the vast majority of people I know of who use it.
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:02
hacky because when the device is on the same network as the server, I think the connection would still cross outside the wlan
Daniel Kumor
@dkumor
Oct 16 2016 18:03
it's also not OSS
and it tunnels the data through their servers
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:04
ah I didn't realize v2 is closed
Daniel Kumor
@dkumor
Oct 16 2016 18:04
Can't give any third parties access to the data at any point, since this is very personal data.
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:05
thanks for pointing that out. I sometimes lose sight of that since I've had to come to terms with the idea of making mine public.
(or is https through 3rd party still a deal breaker?)
Daniel Kumor
@dkumor
Oct 16 2016 18:08
Once again, the only truly safe way to do it is to have a native implementation in the app
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:08
yeah
Daniel Kumor
@dkumor
Oct 16 2016 18:09
Besides, the difficulty of integrating a 3rd party solution is on the same order of difficulty as just implementing it directly
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:10
Probably right. I haven't done any android development, though, so unsure how much I'd be able to help there.
Would you be willing to make a todo list of documentation needs for me?
Daniel Kumor
@dkumor
Oct 16 2016 18:11
sure - but I will only be able to get to it later this week
since the docs have changed a lot
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:11
Ok. Don't sweat it too much...I just figured you'd probably have some things that might be top of mind that you know for sure the docs are wrong/missing for
Daniel Kumor
@dkumor
Oct 16 2016 18:12
Unfortunately, as one of the people who built this thing, I am blind to the issues in understanding that come up
it isn't where I think docs are missing that is important
it is where you think docs are missing that is relevant
but, again, the version of docs currently online is severely outdated, so nothing can be done with that
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:14
Oh I totally get that side of it! I was already going to be approaching them from that standpoint. Was asking so as to definitively take doc tasks you were already aware of off your plate.
the story I'm telling myself is that the next release gets shipped faster if you guys are focusing on the code
Daniel Kumor
@dkumor
Oct 16 2016 18:15
Perhaps an explanation of the roles and permissions? Those seemed to be confusing
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:15
that one's already on my list ;)
Daniel Kumor
@dkumor
Oct 16 2016 18:15
Also an introductory tutorial - from a clean install, what to try out
so how to go from clean install to gathering interesting data
ie, creating ratings, logs, etc
It is the general usage that is lacking docs at this point - the deep technical stuff (pipescript, etc) is covered fairly well.
That's all I can think of right now, but once I publish the current version of docs, I'll create an issue about it, and email you at crawford.comeaux@gmail.com
Is that OK?
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:20
Totally, though I was about to create an issue for what you just mentioned if that works, too.
Daniel Kumor
@dkumor
Oct 16 2016 18:20
sure!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:20
Daniel Kumor
@dkumor
Oct 16 2016 18:20
Yes!
Thanks!
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:21
np!
Daniel Kumor
@dkumor
Oct 16 2016 18:21
BTW: I have to leave now, but I'll try to get the docs up asap
Crawford Comeaux
@crawfordcomeaux
Oct 16 2016 18:22
Right on. question about android app that can wait: is there some means of the app discovering cdb instances on the lan?
That might make it easier to implement the wifi sync if what's specified is the SSID of the wlan, as opposed to an IP.
Thanks for the chat. Adios :)