These are chat archives for Yelp/elastalert

17th
May 2016
Jose Armesto
@fiunchinho
May 17 2016 09:26
any easy way to make elastalert to log the log.error() messages?
—verbose prints INFO messages only
Quentin Long
@Qmando
May 17 2016 16:27
Setting it to info level causes it to print INFO and above, including error
Jose Armesto
@fiunchinho
May 17 2016 16:28
mm… weird :/ I was having some errors, but didn’t see anything on the logs
I’ll check again
Quentin Long
@Qmando
May 17 2016 16:29
Any other details?
Jose Armesto
@fiunchinho
May 17 2016 16:29
It’s some strange behavior related with AWS and signed requests
after a random number of days, elastalert starts failing with 403 Forbidden, because it’s like the http request to fetch ec2 instance metadata to sign the request fails
elastalert doesn’t re-create the elasticsearch object
so it keeps returning 403 forever
in my fork, I changed the handle_error() method, to always set the writeback_es object to None
to see if this helps
Quentin Long
@Qmando
May 17 2016 16:32
Ah. I see. That seems like a reasonable solution
Jose Armesto
@fiunchinho
May 17 2016 16:33
I see that you sometimes set it to None, but not always
any reason?
Quentin Long
@Qmando
May 17 2016 17:57
If there is an error when writing a status document, then there will likely be an error for every subsequent attempt
So we wait until the next cycle of queries until we try again
Quentin Long
@Qmando
May 17 2016 18:12
I'm not sure if thats the most sane strategy, but I did that after flooded with tons of timeout errors, even though elastalert can sometimes run just fine without writeback. Especially true in the case where you have multiple elasticsearch clusters, which I think is not very uncommon, but it's what I was doing