These are chat archives for getredash/redash

19th
Oct 2015
Vipul Nayyar
@vipulnayyar
Oct 19 2015 05:12
Hey there, I need to log what query is finally being sent to mongo by redash. For that, I'd like to log some query data from query_runner/mongodb.py
But using logger.warning or logger.info doesn't add to any of the files in the logs directory
Arik Fraimovich
@arikfr
Oct 19 2015 05:18
@vipulnayyar update the log level (change the value of REDASH_LOG_LEVEL in /opt/redash/.env to INFO or DEBUG) and restart Celery (supervisorctl restart redash_celery or to be on the safe side: supervisorctl restart all).
@vipulnayyar also, make sure you check both /opt/redash/logs/celery.log and /opt/redash/logs/celery_error.log (I think it will be in the later, as this is the one for stderr, where logs are written)
Vipul Nayyar
@vipulnayyar
Oct 19 2015 07:00
Thanks a lot @arikfr :D
Arik Fraimovich
@arikfr
Oct 19 2015 07:00
it worked?
Vipul Nayyar
@vipulnayyar
Oct 19 2015 07:01
yup
Arik Fraimovich
@arikfr
Oct 19 2015 07:01
cool
Vipul Nayyar
@vipulnayyar
Oct 19 2015 08:43
Hey @arikfr , the problem I was facing is that I was passing js code in double quotes in redash query for a mongo data source. The js ran fine in mongo I think but it gave me back a string whereas I want a long to be used while querying. To resolve this, I used execjs module in python to evaluate js code in redash only and then converted it to long as per my requirement before sending it to mongo.
So, is this something that can be worked upon and maybe I can contribute it back to the project?
Arik Fraimovich
@arikfr
Oct 19 2015 08:46
can you show an example query? preferably with the mongo data source we on the demo instance -- http://demo.redash.io/queries/387/source#table. if not possible, then just an example query will do
Vipul Nayyar
@vipulnayyar
Oct 19 2015 08:48
"$match":{"city":"MUMBAI",
"updated_on" : {
"$lt" : "temp = new Date().setHours(23,59,00), new Date(temp).setDate(new Date(temp).getDate() - 1)",
"$gt" : "temp = new Date().setHours(00,00,00), new Date(temp).setDate(new Date(temp).getDate() - 1)"}}
The "updated_on" key is stored as long in my db
But I think after the evaluation of the js in query, it gave me back a string
So, I wrote the some code specific to my requirements that evaluated that js and converted it to a number before sending it to mongo
Arik Fraimovich
@arikfr
Oct 19 2015 08:55
I don't think it parsed the JS you enter there as code. @erans what do you think?
Vipul Nayyar
@vipulnayyar
Oct 19 2015 08:56
Ya, redash doesn't do that, it passes the js to mongo, which actually parses the js
But I guess since the js code is in double quotes, mongo might be giving me the timestamp as string, which I don't want
So I forcefully evaluated the js in redash with execjs module and typecasted to number in python
So, is this a generic scenario that can be worked on and maybe improved in the main codebase?
Arik Fraimovich
@arikfr
Oct 19 2015 08:58
can you show how you added the execjs evaluation?
Vipul Nayyar
@vipulnayyar
Oct 19 2015 09:04
def _fix_dates(self, data):
for k in data:
if isinstance(data[k], list):
for i in range(0, len(data[k])):
self._fix_dates(data[k][i])
elif isinstance(data[k], dict):
self._fix_dates(data[k])
else:
if isinstance(data[k], (str, unicode)):
self._convert_date(data, k)
if k == 'updated_on':
if '$lt' in data[k]:
data[k]['$lt'] = int(execjs.eval(data[k]['$lt']))
if '$lte' in data[k]:
data[k]['$lte'] = int(execjs.eval(data[k]['$lte']))
if '$gt' in data[k]:
data[k]['$gt'] = int(execjs.eval(data[k]['$gt']))
if '$gte' in data[k]:
data[k]['$gte'] = int(execjs.eval(data[k]['$gte']))
Apart from importing execjs
I know this is not very generic solution that can be merged, but it solved my problem
Arik Fraimovich
@arikfr
Oct 19 2015 09:08
definitely need to generalize this.... also need to see if there is no security issue with evaluating with execjs arbitrary code. let's wait for @erans input on this. he uses the mongo datasource, and might have an idea on how to query for relative dates with existing optinos
Vipul Nayyar
@vipulnayyar
Oct 19 2015 09:11
sure :)
anthony
@Sigma-anthony
Oct 19 2015 12:44
hi guys, I would like to run re:dash on docker on marathon. And I am wondering whether to have 3 containers, one with redis, one with postgres and one with redash. Or whether I should run all 3 in the same container?
Or if anyone is running on Docker, could you relate some of thinking that went into structuring the docker container?
Arik Fraimovich
@arikfr
Oct 19 2015 13:29
@Sigma-anthony I suggest to have at least 3 containers as you suggested. You can even split re:dash's container into several ones - one for the web server, another one (or more) for the celery workers. But if you don't have strict requirements, you can start with 3, and use the Dockerfile @johnwu
... and use the Dockerfile @tjwudi created (available in latest master).
anthony
@Sigma-anthony
Oct 19 2015 13:38
thank you @arikfr
Arik Fraimovich
@arikfr
Oct 19 2015 13:50
@Sigma-anthony I'll be happy to hear how it works for you (with docker). Also you will need some proxy like nginx for it.
anthony
@Sigma-anthony
Oct 19 2015 14:25
will do
for now I am having nginx on the same container as redash and separating redis and postgres
so far postgres and redis are running on marathon
working on the creating the tables in postgres so that redash can use it, and I ended up having to clone the redash repo so that I could use the redash module to run manage.py
Arik Fraimovich
@arikfr
Oct 19 2015 14:37
@Sigma-anthony there is a script to create tables in setup/ that you can use a redash image to run. also, you if you check John created nginx config to use with the official nginx container, that you can use alongside his redash container
later this week or early next week there will be official container you can pull from docker hub.
Josh Fyne
@jfyne
Oct 19 2015 14:42
hi, i'm trying to run redash through docker-compose. However I cannot get it running as it appears to be ignoring the REDASH_DATABASE_URL environment variable. Has anyone run into something similar?
Arik Fraimovich
@arikfr
Oct 19 2015 14:51
@jfyne I'll be online later and will be able to help. Can you meanwhile share your docker-compose yml?
Josh Fyne
@jfyne
Oct 19 2015 14:52
sure,
redash:
  image: redash
  ports:
    - "5000:5000"
  links:
    - redis
    - postgres
  env_file: .env
redis:
  image: redis:2.8
postgres:
  image: postgres:9.3
  volumes:
   - /opt/redash-postgres:/var/lib/postgresql/data
i'm using the one from master minus nginx as I am already behind a RP, the defaults in .env.example do not appear to work
with this being the error
OperationalError: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Josh Fyne
@jfyne
Oct 19 2015 15:00
so after seeing that and looking at the redash code I've tried setting REDASH_DATABASE_URL to: postgres://postgres@postgres/redash
this still does not work, however inside the running container with bash I can do the following:
root@569c5d6e8091:/opt/redash/current# env | grep REDASH_DATABASE_URL
REDASH_DATABASE_URL=postgres://postgres@postgres/redash
root@569c5d6e8091:/opt/redash/current# psql $REDASH_DATABASE_URL 
psql (9.3.9, server 9.3.10)
Type "help" for help.

redash=# \dt
             List of relations
 Schema |      Name      | Type  |  Owner   
--------+----------------+-------+----------
 public | activity_log   | table | postgres
 public | alerts         | table | postgres
 public | dashboards     | table | postgres
 public | data_sources   | table | postgres
 public | events         | table | postgres
 public | groups         | table | postgres
 public | queries        | table | postgres
 public | query_results  | table | postgres
 public | users          | table | postgres
 public | visualizations | table | postgres
 public | widgets        | table | postgres
(11 rows)

redash=#
Am I missing something fundamental here?
anthony
@Sigma-anthony
Oct 19 2015 15:06
@arikfr do you mean I can run the bootstrap in a postgres container?
Arik Fraimovich
@arikfr
Oct 19 2015 17:17
@jfyne try running: docker-compose run --rm redash /opt/redash/current/manage.py check_settings. What does it print as database url? Is it possible that when creating the image, you had .env file in your redash folder with different value for DB?
@Sigma-anthony I was referring to this script: https://github.com/EverythingMe/redash/blob/master/setup/docker_init_postgres.sh which you supposed to run in the redash container.
Josh Fyne
@jfyne
Oct 19 2015 17:53
DATABASE_CONFIG = {'name': 'redash', 'host': 'postgres', 'threadlocals': True, 'user': 'postgres', 'password': None, 'port': None}
which suggests to me that it is correct
Arik Fraimovich
@arikfr
Oct 19 2015 17:53
@חכ
Josh Fyne
@jfyne
Oct 19 2015 17:54
let me delete all my images and start fresh
Arik Fraimovich
@arikfr
Oct 19 2015 17:54
@jfyne true. what about the .env file? if you're not sure, try: docker-compose run --rm redash cat /opt/redash/current/.env
Josh Fyne
@jfyne
Oct 19 2015 17:56
it interesting too when you run docker-compose run without an image, it can't find any images to pull
is it just not pushed to the docker repository?
redash:latest that is
Arik Fraimovich
@arikfr
Oct 19 2015 18:00
correct. I will create later this week/early next one, official images that will be available from docker hub
Josh Fyne
@jfyne
Oct 19 2015 18:00
cool, I think I had to make some changes to the Dockerfile that was on master in order to get it to run
Arik Fraimovich
@arikfr
Oct 19 2015 18:01
which ones? I did some too, but nothing major, iirc
Josh Fyne
@jfyne
Oct 19 2015 18:02
aha success, rebuilding the image with the updated .env file worked. So that means that the file passed in with docker-compose doesn't override anything
Arik Fraimovich
@arikfr
Oct 19 2015 18:03
yes, because of the way supervisord starts the processes. I fixed this in a new branch, I just did merge it into master yet
Josh Fyne
@jfyne
Oct 19 2015 18:03
cool well at least you have this covered!
Arik Fraimovich
@arikfr
Oct 19 2015 18:04
oops, I meant did not merge yet :)
anthony
@Sigma-anthony
Oct 19 2015 18:35
@arikfr thats the one I ended up running thanks
Arik Fraimovich
@arikfr
Oct 19 2015 18:46
@Sigma-anthony great. is redash working for you now?
Josh Fyne
@jfyne
Oct 19 2015 18:47
i have also get redash working now
thanks for the pointers @arikfr
Arik Fraimovich
@arikfr
Oct 19 2015 18:48
super!
Josh Fyne
@jfyne
Oct 19 2015 18:51
out of curiosity, in the make file there is this line
cd rd_ui && npm install -g bower grunt-cli

have you ever considered moving those into the package.json as a dev dependency and then running

cd rd_ui && ./node_modules/.bin/bower install
cd rd_ui && ./node_modules/.bin/grunt build

instead?

Arik Fraimovich
@arikfr
Oct 19 2015 18:58
at the time, I think those were the guidelines from bower&grunt. but that was >2 years ago, and things might have changed. I really wish to have the time to upgrade the frontend stack at some point. anyway, if it makes a difference, I can do that.
also, if you aren't planning on developing, you can take the tarballs the build process creates, and then you don't need to built the frontend assets.
Josh Fyne
@jfyne
Oct 19 2015 18:59
i was just curious, it just means that you do not need sudo to install
Arik Fraimovich
@arikfr
Oct 19 2015 18:59
true
eschultze
@eschultze
Oct 19 2015 19:03
Hi! Does anyone is experiencing any trouble using the fabfile.py? I have always used it, but now I'm getting an error
Arik Fraimovich
@arikfr
Oct 19 2015 19:03
@eschultze are you using an updated version?
(compare yours with the latest gist)
eschultze
@eschultze
Oct 19 2015 19:09
ok, updated the script, everything is ok now! thanks @arikfr!
Arik Fraimovich
@arikfr
Oct 19 2015 19:15
:+1: