These are chat archives for getredash/redash

25th
Apr 2016
MikeM
@mmacny
Apr 25 2016 16:53
hey guys - just curous how I enable the schema browser? I've got our Redash connected to presto
Arik Fraimovich
@arikfr
Apr 25 2016 16:55
@mmacny unfortunately it's not implemented for Presto.
MikeM
@mmacny
Apr 25 2016 16:55
Ah, that makes sense :)
Arik Fraimovich
@arikfr
Apr 25 2016 16:56
It should be simple to implement, but I don't have a Presto cluster to test with.
MikeM
@mmacny
Apr 25 2016 17:04
hm, I'll take a look might be pretty quick to drop in
it'll be a huge help to have. also any open-ness to like, a hive metastore browser?
Arik Fraimovich
@arikfr
Apr 25 2016 17:17
@mmacny I'll send you some pointers to the code soon to make it easier.
MikeM
@mmacny
Apr 25 2016 17:17
:) awesome
Arik Fraimovich
@arikfr
Apr 25 2016 17:17
I'm not familiar with Hive, is this the equivalent of a sschema?
MikeM
@mmacny
Apr 25 2016 17:18
Yeah, so if you think of Hadoop as basically a giant filesystem... Hive has the ability to execute queries against all of that (treat it like a DB), but Hive uses a Database of pointers etc to know where all that junk lives
(i.e. the Metastore). So it's got things like partition structure and location
One nice thing about Hue (a browser-based query tool for Hadoop) is that you can browse that metastore, and also walk the Filesystem.. Helpful to know if data exists
Redash has easily outpaced all the integrations of all the competitors, so having some of these little nice-to-haves would def. make it out one stop tool
Arik Fraimovich
@arikfr
Apr 25 2016 17:23
Sounds like something I'll be happy to see in Redash
MikeM
@mmacny
Apr 25 2016 17:24
awesome!
Arik Fraimovich
@arikfr
Apr 25 2016 17:24
I'm not sure the current ui will be good enough for the Metastore but I'm planning an upgrade for it as well.
If/when you get to it and see it's hard to represent the Metastore with current ui, let's talk.
Arik Fraimovich
@arikfr
Apr 25 2016 17:33
@mmacny this is the query runner implementation of Presto: https://github.com/getredash/redash/blob/master/redash/query_runner/presto.py
You need to implement the get_schema method which returns an array of dictionaries with name field (table name) and columns array with column names.
MikeM
@mmacny
Apr 25 2016 17:42
works for me. thx for the pointer
Arik Fraimovich
@arikfr
Apr 25 2016 18:12
:+1:
@mmacny I've read your blog post about the move to Presto. Something that wasn't clear to me - are you querying Redshift via Presto as well?
MikeM
@mmacny
Apr 25 2016 18:14
Occasionally. We're actually just migrating most stuff away from Redshift in general and trying to have tools that don't require specialized sources
Still in transition. Presto's kinda nice for migrations bc we can join legacy tables (in redshift) to current tables in Hive
and, if the query is slow, we just add nodes mid-query
Arik Fraimovich
@arikfr
Apr 25 2016 18:16
Interesting. And you find Hive's performance good enough to move from Redshift?
MikeM
@mmacny
Apr 25 2016 18:16
Eh, apples and oranges. Redshift is lightning fast. Hive at least gives us all the data at our fingertips, rather than just the fraction that we've decided to put into Redshift
so if you throw Presto atop Hive, it's not a terrible slowdown
Arik Fraimovich
@arikfr
Apr 25 2016 18:18
Got it. So you going to keep both and use Redshift for the smaller data sets or moving completely to Presto?
MikeM
@mmacny
Apr 25 2016 18:19
yep. For our clients we'll use stuff like ElasticSearch to keep it in semi-raw form and provide flexibility... but for our teams they are able to take care of a lot of their needs with Presto
and multiple catalogs being involved in the same query is pretty rad
Jeff Widman
@jeffwidman
Apr 25 2016 18:19
hey @arikfr working through setting up additional users today
how do I lock it down to all users on a particular google apps domain?
I'm using the docker images
the other thing I was going to do today was put together a PR for how to use url query params in queries/dashboards
should I just stick that under the
"usage" topic?
Arik Fraimovich
@arikfr
Apr 25 2016 18:22
@mmacny cool. Looking forward to read more about your experience with moving to Presto from Redshift. Specially from the usage perspective.
@jeffwidman there is a cli to add a domain. It's mentioned in the setup doc.
@jeffwidman under usage sounds roght
*right
Thanks!
Jeff Widman
@jeffwidman
Apr 25 2016 18:24
Is the google oauth required for the domains for google apps?
or only if you want to let anybody use google oauth regardless of domain?
Arik Fraimovich
@arikfr
Apr 25 2016 18:25
Nope, required.
Jeff Widman
@jeffwidman
Apr 25 2016 18:25
thx
Jeff Widman
@jeffwidman
Apr 25 2016 20:09
started working on the query params PR, and just kept feeling like the docs overall structure could use some tidying up
feedback from teh community?
getredash/redash#1012
Arik Fraimovich
@arikfr
Apr 25 2016 21:20
I like it ! :+1: