These are chat archives for getredash/redash

23rd
Feb 2016
Scott Reynen
@sreynen
Feb 23 2016 15:59
i'm getting random ��������������� characters in built JS files whenever i change anything. questions: 1) any idea what causes that? 2) where do the built JS files reside? 3) is gulp build the correct way to build them?
Scott Reynen
@sreynen
Feb 23 2016 16:05
or more generally: is there any documentation on the build process?
Scott Reynen
@sreynen
Feb 23 2016 16:11
i see a file in rd_ui/dist/scripts/embed-scripts.1662c8ec.js that looks like it contains the file i'm editing, but it doesn't contain those random characters, and the path in my browser is completely different, /scripts/directives/dashboard_directives.js
Rivkah Standig
@rivkahstandig3636
Feb 23 2016 16:17
even more general question: is there any documentation on contributing to redash, the code base, how to set up dev environments? @sreynen and i have figured most of it out by trial and error but just wondering if there's some docs somewhere that we aren't finding :)
by "the code base" i meant navigating the code base
Arik Fraimovich
@arikfr
Feb 23 2016 16:26
Unfortunately not :-( I'll be happy to create such document if you send over a list of questions that will be the basis for it.
Build process: the closest thing to documenting it is the circle.yml file which the ci use.
But can you explain how you setup your Dev env? Will help me understand what issues you might be encountering.
And last note - for development you don't need to build the assets. You can use the"raw" ones.
(sorry for brevity, typing from my phone while cooking 🍳)
Scott Reynen
@sreynen
Feb 23 2016 16:29
we started with basic ubuntu/trusty64 vagrant box, and ran bootstrap.sh
then changed /opt/redash/current to point to a current git clone
then restarted supervisord, which broke server things, so restarted the vagrant box, which fixed server things. then discovered python packages needed to be added, which got the server running. then figured out bower install was necessary to get the UI working correctly
Rivkah Standig
@rivkahstandig3636
Feb 23 2016 16:32
how can we use the raw ones? can you explain how to get the ui to update with our changes if we don't build the assets each time we change them?
Arik Fraimovich
@arikfr
Feb 23 2016 16:43
Yes. It just a few configuration changes. The bootstart script is intended for "production" so has different configuration than development. I'll send the instructions later today.
eschultze
@eschultze
Feb 23 2016 16:43
Hi, can we use API with parameter queries?
Scott Reynen
@sreynen
Feb 23 2016 16:48
https://github.com/getredash/redash is where i started reading about how to contribute, and i would expect that's where most people start. there's only one link on that page to how to get an instance running, so i started there and picked what looked like the most dev-friendly version, http://docs.redash.io/en/latest/setup.html#other
Rivkah Standig
@rivkahstandig3636
Feb 23 2016 16:51
thanks @arikfr! that would be a great thing to add to the docs :)
Arik Fraimovich
@arikfr
Feb 23 2016 19:14

@rivkahstandig3636 @sreynen all Re:dash's settings are in redash/settings.py, which has default values and reads overrides from environment variables. The way the bootstrap.sh script sets things up is that the settings are in /opt/redash/.env (symlinked to /opt/redash/current/.env) which is read before starting the process by bin/run.

The setting that defined if to use the built assets or the raw ones is REDASH_STATIC_ASSETS_PATH.

This is about the setting. Now let's review what's needed to run and develop Redash:

Redash has the following dependencies:
  1. Python 2.7
  2. PostgreSQL 9.3
  3. Redis 2.8
  4. Node.js (only in development for building the static assets and getting bower packages)
  5. Some system packages, which depend on 1) the data sources you want to use; 2) the operating system you use. You can deduce what might be needed from the Dockerfile or bootstrap.sh.
For development it's enough to install packages from requirements.txt and requirements_dev.txt (pip -r requirements.txt -r requirements_dev.txt).
You can run tests with: make test.
To install the frontend stuff, run npm install and npm run bower install from rd_ui.
You can run npm run build to build minified assets, but it's not needed for development.
Arik Fraimovich
@arikfr
Feb 23 2016 19:19
To create the database tables, run: ./bin/run python manage.py database create_tables (you might need to create the database first with psql).
And then run ./bin/run python manage.py users create --admin --password admin "Admin" "admin" to create the first user.
To run the system you need two processes:
  1. Web server: bin/run ./manage.py runserver
  1. Celery workers: ./bin/run celery worker --app=redash.worker --beat -Qqueries,celery,scheduled_queries
And.... you're done :)
Questions?
eschultze
@eschultze
Feb 23 2016 20:07
Hello people! Does anyone know how to use API with parameter queries?
Arik Fraimovich
@arikfr
Feb 23 2016 20:17
@eschultze I did some work on that yesterday but not everything is possible yet. What do you want to do?
eschultze
@eschultze
Feb 23 2016 20:27
Hi @arikfr, something like .../api/queries/21/results.csv?p_ip=8.8.8.8&api_key=mykeymykeymykey
I'm getting {"message": "Internal Server Error"}
using curl
Scott Reynen
@sreynen
Feb 23 2016 20:40
@arikfr that's a great start! my REDASH_STATIC_ASSETS_PATH is set to ../rd_ui/dist/, but the files in there don't match what i'm seeing in my browser. is there some sort of source mapping going on?
oh, maybe that's not actually my REDASH_STATIC_ASSETS_PATH, only the value in /opt/redash/.env, which maybe isn't being read
it looks like it's using the default value, ../rd_ui/app/. but it's not updating live. when i change those files, i eventually see the changes, but not immediately and i don't know what is causing the lag
Scott Reynen
@sreynen
Feb 23 2016 20:49
i've confirmed my browser cache is disabled and restarted redash_server
but i'm still seeing a stale version of my files
Arik Fraimovich
@arikfr
Feb 23 2016 20:53
@sreynen yes if the .env file isn't linked to /opt/redash/current, then it's not used. the delay might be due to caching of static files. This caching is disabled when running the server in debug mode (which isn't happening in your case).
@eschultze the error is for another reason, but it won't work anyway. Keep following the updates in the project, soon I'm going to add support for this.
Scott Reynen
@sreynen
Feb 23 2016 20:54
it is possible to manually clear that cache? and what controls debug mode?
i don't see any mention of debug in settings.py
eschultze
@eschultze
Feb 23 2016 20:55
@arikfr ok thanks ;)
Arik Fraimovich
@arikfr
Feb 23 2016 21:04
@sreynen yes, clear browser cache. Debug mode is a parameter you pass to "manage.py runserver" but in your case youre not even using manage.py . I suggest you stop supervisord and run processes manually with the commands i posted above.
Scott Reynen
@sreynen
Feb 23 2016 21:06
@arikfr so there's no server-side cache of JS files?
my browser cache is off
it doesn't make any sense to me that the file in my browser and the file on the server are not the same. there must be something in between
i guess i'll make a new VM from the redash box. i don't know how to undo everything bootstrap did
Arik Fraimovich
@arikfr
Feb 23 2016 21:12
Maybe it is somehow running the code that was previously in the current folder. You can either reinstall or just stop supervisord.
Scott Reynen
@sreynen
Feb 23 2016 21:13
no, i know it's running the code i'm editing. the versions are different
"just stop supervisord" is a thing i can probably figure out how to do, but not really how i want to spend my time
Arik Fraimovich
@arikfr
Feb 23 2016 21:20
:) sudo /etc/init.d/redash_supervisord stop
Scott Reynen
@sreynen
Feb 23 2016 21:21
i'm fairly sure it restarts again when i restart the box
Arik Fraimovich
@arikfr
Feb 23 2016 21:21
right. Delete this init file to prevent thst as well
I am going to sleep. I will answerr more questions in the morning
Scott Reynen
@sreynen
Feb 23 2016 21:22
i don't want to jump over these barriers to entry. i want to lower them
Scott Reynen
@sreynen
Feb 23 2016 21:35
using the vagrantfile in the repo seems to have fixed all my problems. i made a PR to suggest that in the contrib instructions
cldellow
@cldellow
Feb 23 2016 22:35
hey sreynen, did you just get the redash dev environment working?
Scott Reynen
@sreynen
Feb 23 2016 22:36
yes
cldellow
@cldellow
Feb 23 2016 22:36
can I ask you a few questions? :) i'm just setting it up now via vagrant + the vagrantfile in the repo
Scott Reynen
@sreynen
Feb 23 2016 22:37
sure
i won't necessarily know the answer, but ask away!
cldellow
@cldellow
Feb 23 2016 22:37
heh, fair enough. after following the directions, i launch redash via bin/run honcho start -f Procfile.dev
and sure enough, i can go to http://localhost:9001 and see the readsh login UI
when i login with admin/admin, I get a strack trace of "ValueError: invalid literal for int() with base 10: 'default'"
i haven't started digging into it too much, figured I'd ask and see if there's some step missing in getting the clean environment working
i think it's some config for finding the metrics db is interpreting a port of "default" as an integer rather than putting in something sane like 5432
Scott Reynen
@sreynen
Feb 23 2016 22:44
you have a file at /opt/redash/current/.env ?
cldellow
@cldellow
Feb 23 2016 22:45
yeah, I have the default one copied from /opt/redash/.env
Scott Reynen
@sreynen
Feb 23 2016 22:45
i notice the default value for REDASH_DATABASE_URL is different from the value in that file, so maybe it's trying to connect with the wrong database URL
specifically, the default value is postgresql://postgres and the value in that file should be postgresql://redash
cldellow
@cldellow
Feb 23 2016 22:46
yeah, I think that's OK
I do have a redash DB and when I log into it with psql, it looks like it has the right tables
it's also not clear that REDASH_CONNECTION_STRING gets used anywhere - doesn't show up if I grep for it
Scott Reynen
@sreynen
Feb 23 2016 22:48
i don't know about that one, but REDASH_DATABASE_URL is used in redash/settings.py
cldellow
@cldellow
Feb 23 2016 22:49
yeah, I believe it's correct. If I change it to point to an invalid db (like redashfoo), I get an exception about that being a nonexistent database
Scott Reynen
@sreynen
Feb 23 2016 22:49
hmm, that's my only guess. should be identical environments
cldellow
@cldellow
Feb 23 2016 22:52
did you do the step where it talks about applying migrations?
when I do that, I get a ton of noise/error messages. unsure if they're intentional or not.
Scott Reynen
@sreynen
Feb 23 2016 22:53
i did, but i don't remember what the output was
and it's long gone now in the flood of logging
cldellow
@cldellow
Feb 23 2016 22:54
no worries, i'll check back in here later. thanks for the help!
oh, hmm, that step to run migrations is a bit busted. it doesn't ensure they run in the correct order. :P
Scott Reynen
@sreynen
Feb 23 2016 22:59
one thing i did when i was trying to do a more manual install (a process i don't recommend in general, but this part might be useful) is manually drop the redash.public schema and then do sudo -u redash bin/run ./manage.py database create_tables and sudo -u redash bin/run ./manage.py users create --admin --password admin "Admin" "admin"
which i got from bootstrap.sh
cldellow
@cldellow
Feb 23 2016 23:03
that might get me past this. I'm pretty sure the issue is out of date DB schema/data. the find command to get the list of migrations to run is broken (doesn't guarantee ordering) - but even if you fix that, you still see the same errors. i'll try recreating the db via those commands
cldellow
@cldellow
Feb 23 2016 23:09
huzzah, that works
Scott Reynen
@sreynen
Feb 23 2016 23:12
great
i'm trying to figure out where dashboard widget rendering happens, if anyone can help with that
cldellow
@cldellow
Feb 23 2016 23:45
thanks again, @sreynen! hope you have luck figuring out the dashboard widget stuff.
Scott Reynen
@sreynen
Feb 23 2016 23:46
no problem. think i'm getting close. angular is new to me