These are chat archives for Yelp/elastalert

10th
May 2016
Alexander Olsson
@noseglid
May 10 2016 06:47
Hello,
Do you maintain a changelog for elastalert version?
Marcus Carlsson
@xintron
May 10 2016 11:21
Is it possible to set the smtp_{host,port,...} in the global config so that any rule using email alerts will use the same default config?
Quentin Long
@Qmando
May 10 2016 16:49
@noseglid No changelog right now, sorry. Curious about anything specific?
@xintron Yes, that should work fine
Alexander Olsson
@noseglid
May 10 2016 17:38
@Qmando Nah, we had an issue where it seemed elastalert all of sudden did all queries with a 24 hr timeframe, even though our alerts are defined at between 30 min and 1 hr (frequency and flatline types)
So we have above e.g. 500 hits of one query the last 15 minutes should generate an alert, but it did 500 hits the last 24 hrs. Unable to reproduce it so can’t give more info, unfortunately.
Quentin Long
@Qmando
May 10 2016 18:06
@noseglid If you stop ElastAlert for 24 hours and then restart it, the first query it will make will be for that entire period since it last run. The same would happen if Elasticsearch started returning 503s or something for a while. That's the only thing I can think that would cause that
Alexander Olsson
@noseglid
May 10 2016 18:15

It happened over and over again, We had it running every 15 minutes, and every time it would send an email saying something like “between 2016-05-09 08.00 to 2016-05-10 08.00 there were more than 500 events”. That is a whole days span.

After restarting it went away, but not without the on-call person waking up and sweating :)

We also upgraded to 0.0.80 on the offchance it was fixed.
Quentin Long
@Qmando
May 10 2016 18:16
Oh, so your timeframe is 24 hours
It just prints out 24 hours range no matter when the actual events were
Alexander Olsson
@noseglid
May 10 2016 18:19
That’s not true. E-mails before didn’t do it
At least 500 events occurred between 4-18 6:26 UTC and 4-18 7:26 UTC
That’s from an e-mail a few days ago
Quentin Long
@Qmando
May 10 2016 18:20
Were they nottype: frequency? Did they have alert_text_type: alert_text_only?
And your timeframe didn't change at all?
Alexander Olsson
@noseglid
May 10 2016 18:21
yea, type: frequency, alert_text_type is not defined at all
timeframe is hours: 1
Quentin Long
@Qmando
May 10 2016 18:22
That line is just most recent event minus timeframe. https://github.com/Yelp/elastalert/blob/master/elastalert/ruletypes.py#L231
wait. I was reading that first line wrong
The text was different? Like one was local time, one was raw timestamps?
This seems very strange indeed
Alexander Olsson
@noseglid
May 10 2016 18:26
Yea :( If we see it again I’ll get back