These are chat archives for getredash/redash

6th
Sep 2016
HenroRitchie
@HenroRitchie
Sep 06 2016 02:20
@arikfr Hi. How did you create the AWS image? Did you define any swapspace?
@arikfr I see from you earlier reply you don't use any swap space. It looks like (and I will have to confirm) my issues with AWS was related to OOM killing the redis server and the solution was to define swapspace. I did read that the nano instance and the micro instance both get swap space allocated to them by AWS but the larger instances don't.
Arik Fraimovich
@arikfr
Sep 06 2016 05:11
@HenroRitchie I didn't actively define any swapspace, and it's something I usually avoid on my servers. Nothing good ever comes out of it :)
@sreynen you can use the refresh api and to pass it parameters.
HenroRitchie
@HenroRitchie
Sep 06 2016 06:40
@arikfr Noted. It is something I stumbled upon somewhere in troubleshooting redis.
Arik Fraimovich
@arikfr
Sep 06 2016 06:42
@HenroRitchie what instance size are you using?
HenroRitchie
@HenroRitchie
Sep 06 2016 06:43
@arikfr AWS Medium. 4G memory and no swap file.
Arik Fraimovich
@arikfr
Sep 06 2016 06:44
Usually that's plenty enough. Did you see what consumes the memory? Do you have very large result sets?
HenroRitchie
@HenroRitchie
Sep 06 2016 06:44
@arikfr AWS Nano and Micro both have swap space built into the instance, but not the larger ones.
I do have quite large datasets, a 1000 records gets refreshed every minute
And when I try to do 10000+ records I quickly run into problems.
Arik Fraimovich
@arikfr
Sep 06 2016 06:46

1000 rows isn't large (although it also depends on how many columns you have).

But maybe the problem is Celery's results caching? If Redis consumes most of the memory, you can adjust for how long Celery stores the results. I think the default is a few hours, but you can reduce it to a few minutes.

HenroRitchie
@HenroRitchie
Sep 06 2016 06:48
Ok, let me have a look at that. A 1000 might not be many, but redis only seems to have problems after a couple of hours, say 12. Basically every morning I first have to check if OOM killed redis during the night
Arik Fraimovich
@arikfr
Sep 06 2016 06:48
What version of Redash do you use?
And how much memory does Redis take?
HenroRitchie
@HenroRitchie
Sep 06 2016 06:48
0.11.1+b2095
The first redis memory writes start at about a 100meg, thereafter it grows by about 20 every time
Arik Fraimovich
@arikfr
Sep 06 2016 06:49
that's unusual. I would try adjusting the Celery cache settings.
once you do that see if it stops growing (as new results will "take the place" of old ones)
HenroRitchie
@HenroRitchie
Sep 06 2016 06:51
Ok. Will do, for now I have to run to the factory. Will feedback later
Arik Fraimovich
@arikfr
Sep 06 2016 06:51
have a great day :)
HenroRitchie
@HenroRitchie
Sep 06 2016 06:52
Will do, same to you.
Arik Fraimovich
@arikfr
Sep 06 2016 20:00
@HenroRitchie this is the setting I was referring to : http://docs.celeryproject.org/en/latest/configuration.html#celery-task-result-expires