These are chat archives for DataBrewery/cubes

5th
Jul 2016
Steve Adams
@stevezau
Jul 05 2016 08:39
anyone
?
Jose Juan Montes
@jjmontesl
Jul 05 2016 09:43
As of today Cubes sends queries directly to the backend database. It is planned to add pre-aggregation capabilities to next versions.
Steve Adams
@stevezau
Jul 05 2016 10:03
ah ok
is it in 1.2?
sorry, planned for 1.next?
thanks for responding @jjmontesl
Jose Juan Montes
@jjmontesl
Jul 05 2016 10:15
Planned for 1.2/2.0 (don't know exactly how the version will be named)
But there is no code you can test yet. It's on the drawing table.
Steve Adams
@stevezau
Jul 05 2016 10:17
ah ok.. is it a high priority item? Just trying to understand expected timeframes
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:06
It's one of the main reasons
Sorry, one of the main targets for next release.
Timespan.... I cannot tell. The task relies mostly on Stefan, but I'd say 2-6 months...
The few required model changes are already being worked on, most of the discussion resolved. But you know, requires some refactoring and it's being worked on on spare time.
Steve Adams
@stevezau
Jul 05 2016 13:09
ok great
good to know
it will be important for me down the track so just making sure it's something on the roadmap
thanks P)
:)
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:10
:)
btw, stick to 1.1.x for now, master is already undergoing breaking changes
Steve Adams
@stevezau
Jul 05 2016 13:10
i'm very new to cubes/olap.. i'm reading the documentation but finding it a little hard to follow
ok
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:10
what's your use case?
Steve Adams
@stevezau
Jul 05 2016 13:10
do you have any other tutorials or things i can look into?
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:11
there are some materials online, but beware some are a bit obsolete
There are a lot of resources for OLAP / star schema warehousing.
Steve Adams
@stevezau
Jul 05 2016 13:12
i've already built a little BI app that pulls in data from Mindbody (POS), Deputy (timesheets) and Xero (Accounts).. it provides my business some dashboards and insights into how thigns are running.. It's starting to slow down with all the number crunching going on.. i've pretty much written a python reporting module on the app.. I had no idea about data cubes and think it could greatly simplify the app
its python backend w/ angular frontend.. im thinking angular could just query for the data directly using OLAP..
it's pretty much just a sql database with a bunch of tables
pretty simple really
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:13
Cubes is very lightweight and simpler to configure and use (though less expressive for some queries). You mostly need to have a data warehouse database and configure your cubes model. I recommend CubesViewer to explore your cubes (at least during development) -disclaimer: I'm the author-.
My fork of cubes-examples has a working model and data for 1.1 that can serve as example: https://github.com/jjmontesl/cubes-examples
Steve Adams
@stevezau
Jul 05 2016 13:14
data warehouse database.. i'm really new to this area.. Is that just the way you define the schema or is it software driven?
oh, ok great
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:16
Cubes allow for a particular kind of queries. A full OLAP database is less flexible than a OLTP(SQL) database. Cubes, being simpler, is yet a bit less expressive (may be limited for complex queries), but it has served me well (most of the shortcomings can be worked around with some data transformation).
data warehousing database: although cubes can pretty much map any database schema as a "dimensional" model (required for OLAP), there are some usual database strategies.
The normal pipeline is:
1: Data source(s) -> 2: Data transformation (ETL) -> 3: Data warehouse database
Steve Adams
@stevezau
Jul 05 2016 13:18
hmm i think i'll need to do some reading
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:19
The data warehouse database (aka "data mart") can be anything, but in many cases it is a relational database with a particular schema (Star Schema / Snowflake Schema). This is called Relational Backed OLAP (ROLAP). A nice thing of Cubes is that it allows you to map any database schema, not forcing you to do any transformation for many cases. So it is an option for you to simply map your existing database.
I personally always run a data transformation process (simple python scripts) and load a database instance, which then I use from Cubes.
Steve Adams
@stevezau
Jul 05 2016 13:20
yeah i think i understand it. I have the opportunity to redeisng the database at the moment so i think it's a good idea i do some research on star schema / snowflake schema (as you suggested)
well, i have full control over the data..
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:20
Note, however,that your data-warehouse schema is separated from your normal transactional schema.
Steve Adams
@stevezau
Jul 05 2016 13:20
oh
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:21
I recommend not to mix them, and completely ignore OLAP when designing your transactional database. Design your database as usual (normal forms, denormalization according to your intuition/experience, etc...).
Steve Adams
@stevezau
Jul 05 2016 13:22
the thing is.. this app is really to display reports on the data
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:22
Then extract data for analysis and load a separate instance which you use for OLAP. In these kind of processes you have the opportunity to merge other information from other systems (ie country population, data from other tools, etc).
Steve Adams
@stevezau
Jul 05 2016 13:22
it has no need to process transactions etc.. so i think a data warehouse is what I need
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:22
Ah, possibly :)
If it's an integration of Business Intelligence, then gather all your data, write your import processes which you already probably have. Using a Star Schema/Snowflake schema shall be appropriate. Model thinking of dimensions and facts. Define your model and you shall be ready to go.
Steve Adams
@stevezau
Jul 05 2016 13:24
sounds easy.. :P
i think i might also look into SaaS BI tools rather then building my own :|
before i go down the rabbit gole
hole*
thanks for your help.. so far :P
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:29
There are a few very good ones. I think we are not there yet :). In my experience depends on whether you'll be doing a lot of transformation anyway.
Steve Adams
@stevezau
Jul 05 2016 13:30
what's the good ones your experience?
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:30
SaaS tools have connectors for a lot of stuff, but the more custom data you have, the more troublesome is to use them.
Steve Adams
@stevezau
Jul 05 2016 13:30
i don't think they will work for me but it's worth looking before i dive in
yeah i know.. i work for SumoLogic.. we do machine data analyics.. it's a bit different but same sort of challanges
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:31
I haven't been a customer of any of them, just checked their features and online videos/tutorials. So have a look yourself, I don't want to clutter this room with "the others" mwaahaha ;)
Steve Adams
@stevezau
Jul 05 2016 13:31
my wife has a little shop which is what im building this bi app for :P
haha no worries
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:33
No, seriously, cannot really opinate, there are a few from the top of my head but all are different in some way... Microstrategy, Datawrapper, Raw, Plotly, Tableau, QLik.
Steve Adams
@stevezau
Jul 05 2016 13:34
yeah i'm aware of those.. and their marketing fluff :P
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:34
There is the famous full-stack open-source business-intelligence suite Pentaho. Java based, clients a bit obsolete. But it is monolithic, the opposite of what we try to build, and zero pythonic.
Steve Adams
@stevezau
Jul 05 2016 13:34
i hanged up my java hat a long time ago :P
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:36
I have an experimental and mostly undocumented ETL tool (able to create and load star/snowflake schemas) that I wouldn't recommend for production purposes, but in case you feel like contributing to an OSS project and you are interested... https://github.com/jjmontesl/cubetl
For a little shop, go with cubes
Steve Adams
@stevezau
Jul 05 2016 13:36
i think that's the plan
got some research to do
i gota run, thanks once again
Jose Juan Montes
@jjmontesl
Jul 05 2016 13:38
Glad to chat. Have fun!