These are chat archives for getredash/redash

3rd
May 2018
Ramzi
@cheframzi_twitter
May 03 2018 03:12

Hi I have a question we are on version 3.0.0+b3134 of redash and have been having alot of issues with the celery worker crashing when a redash user uses high IO type of query against redshift. All we see in the logs is this

[2018-05-03 02:56:50,907][PID:21373][ERROR][MainProcess] Task redash.tasks.execute_query[b0a5459a-fb6e-4d47-891f-d2ee64444539] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1175, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).

but then all the other workers and the box go to 100% cpu and 100% memory
Our supervisord config is standard

[program:redash_celery]
command=/opt/redash/current/bin/run celery worker --app=redash.worker --beat -c10 -Qqueries,celery --maxtasksperchild=10 -Ofair
directory=/opt/redash/current
process_name=redash_celery
user=redash
numprocs=1
autostart=true
autorestart=true
stdout_logfile=/opt/redash/logs/celery.log
stderr_logfile=/opt/redash/logs/celery_error.log

The ask: Anyone here seen an issue, are we configuring things wrong?

but unfortunately ONE query is ruining the experience of everyone. So I am wondering if there is a way to kill the worker and remove the query that caused the issue this way the system as a whole does not become unstable
Renato Marinho
@renatomarinho
May 03 2018 04:57
This message was deleted