These are chat archives for connectordb/connectordb

17th
Oct 2016
Daniel Kumor
@dkumor
Oct 17 2016 02:19
Unfortunately, not that I am aware of. At least for this version, the plan is to use direct IP.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 18:54
I just created a log for my daily todo lists & it occurred to me it'd be nice to use github-flavored markdown in them, but if the data's stored simply as that text, it'd need to be cleaned if I wanted to do any kind of NLP analysis later. Might it make sense to map markdown to some sort of json structure to store alongside the text?
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 19:14
Is that something pipescript might be appropriate for?
Daniel Kumor
@dkumor
Oct 17 2016 19:16
Yes, pipescript would be the correct way to clean up text
no such function exists at this time, though
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 19:23
I'll dig into pipescript later today & get a feel for it. Do you have an idea of pipescript's scope, ie. some way to determine if adding to the language makes sense?
Daniel Kumor
@dkumor
Oct 17 2016 19:28
Adding functions is totally fine - changing the fundamental way things are parsed is much less desirable - or at least it is not anywhere near my personal list of priorities. As it stands, I think it performs the function it was built for without much issue. I believe that I only have one planned function left to add before it is ready for the next release.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 19:33
Cool. Thanks for the guidance :)
Daniel Kumor
@dkumor
Oct 17 2016 19:34
no problem
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:11
are stream datatypes mostly there as a placeholder for use in custom transforms to filter streams?
Daniel Kumor
@dkumor
Oct 17 2016 20:13
stream datatypes are currently used only for display purposes - the datatype tells ConnectorDB what the data represents, so it can be plotted correctly. Ie, "rating.stars" shows you stars when adding to the stream.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:25
So there's a set of predefined datatypes that impacts how the data's displayed. Users can define new ones, but that serves no purpose unless the user extends the front-end to support the new ones?
Daniel Kumor
@dkumor
Oct 17 2016 20:26
While it serves no official purpose currently, the idea was to allow these datatypes to be hints towards machine learning
so the datatype tells a learning algorithm how to interpret it
Each datatype would go to increasing granularity separated by .
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:28
Ooooh that's good to know. Are there other features that are waiting for future purpose like that? (eg. I don't know how I'll be applying ML to video/image streams of me through my day, but I'm sure that one day I or others will find a use for it, so I'm collecting the data now)
Daniel Kumor
@dkumor
Oct 17 2016 20:29
That is the main "future-oriented" field - a couple others (like isVisible) were originally also meant for ML, but found a use in the display frontend
meaning machine learning algorithms would have streams of their own, which would be hidden from view
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:30
Ok. So we have a basic naming convention.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:44
This might be a good time to ask who the target audience is for CDB. If it's for devs, then describing how to add new creators/inputs would make sense. If it's for non-devs, it'd make more sense to just give the basics on datatypes & maybe plan a UI for building creators/inputs.
Daniel Kumor
@dkumor
Oct 17 2016 20:47
The hope is to have core functionality that "just works", with a clear path towards advanced usage for technical people.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 20:51
Gotcha.
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 21:09
It may not be mature enough for now, but this looks like a really nice way to simplify dataviz https://vega.github.io/vega-lite
Daniel Kumor
@dkumor
Oct 17 2016 23:52
The most recent documentation is online at https://dev.connectordb.io/docs/0.3.0b1/
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 23:53
Woo!
Daniel Kumor
@dkumor
Oct 17 2016 23:53
You will see that the new site/docs are very much a work in progress, but it will give an idea of all the missing parts to the docs
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 23:55
So that's the current deployment of https://github.com/connectordb/connectordb.github.io?
Daniel Kumor
@dkumor
Oct 17 2016 23:55
it is a deployment of the v0.3.0b1 branch
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 23:56
just clicked branches & saw it :D
Daniel Kumor
@dkumor
Oct 17 2016 23:56
it will auto-update on push, so this will always be the most recent version
Crawford Comeaux
@crawfordcomeaux
Oct 17 2016 23:56
rockin