Thanks for the info @Qmando I'll get going with the stuff..
Hello - I am looking at making lots of rules that looks similar (for example - check number of RC 401 for a set of end points). Are there any suggestions as to what is the best way to do this ? Should it be one rule per end point, or one rule for all end points? I am assuming that multiple rules = multiple queries... is there a way to consolidate the results and parse ?
@ying1 You can do all end points in a single rule
There are several features that support this. You can use
query_key which will group documents by certain key and then count them separetely. So for example, if you have frequency type with num_events: 10 plus query_key: hostname, it would only alert if a single hostname got more than 10 events
Query_key also works for other types, basically just applies the log separately for each unique value of that field (or combo of yields)
There's also
use_terms_query which will make a terms aggregation query instead of normal query if there are too many documents
You should use a virtualenv if you are not already
pip install elastalert will try to install all of the latest version of each dependency. If you
pip install -r requirements.txt, it will use a known working (and compatible) list dependencies, though some of those versions are pretty old
If you are getting compile errors for blist, that's indicative of a messed up pip installation
Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-5RHl3s/blist/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-6lrisv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-5RHl3s/blist/
pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)
version ^
I get the above error when I try and run pip install elastalert as root
When I run python setup.py install I get the following error Running blist-1.3.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xDDIzl/blist-1.3.6/egg-dist-tmp-PPWuCk
warning: no files found matching 'blist.rst'
blist/_blist.c:38:20: fatal error: Python.h: No such file or directory
include <Python.h>
^compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
I think this a PiP issue but I dont know how to fix it yet.
Traceback (most recent call last):
File "/bin/elastalert-create-index", line 11, in <module>
load_entry_point('elastalert==0.1.2', 'console_scripts', 'elastalert-create-index')()
File "/usr/lib/python2.7/site-packages/elastalert-0.1.2-py2.7.egg/elastalert/create_index.py", line 116, in main
if es_index.exists(index):
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, *kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 212, in exists
self.transport.perform_request('HEAD', _make_path(index), params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 67, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ConnectionError(('Connection aborted.', error(111, 'Connection refused')))
File "/bin/elastalert-create-index", line 11, in <module>
load_entry_point('elastalert==0.1.2', 'console_scripts', 'elastalert-create-index')()
File "/usr/lib/python2.7/site-packages/elastalert-0.1.2-py2.7.egg/elastalert/create_index.py", line 116, in main
if es_index.exists(index):
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, *kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 212, in exists
self.transport.perform_request('HEAD', _make_path(index), params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 67, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ConnectionError(('Connection aborted.', error(111, 'Connection refused')))
@Qmando Coming back to my question of Sep 8 :smile: If I want to alert when there are no matches, the flatline only works if there was a match first and then drops to 0 in for example 24 hours. I want to alert with 0 matches. Custom ruletypes/alerters/enhancements won't work as they are all seem to be only triggered when matches occur. Is there any way to specify and alert when there are no matches? (example usecase "virusdefinitions not updated", it's easy to find out with 1 query if some line about updating occurs...if it's 0 times in the last 24 hours and alert has to be sent...we cannot expect that there ever was a succesful line about updating beforehand)
Hello - have a question relating to monitor frequency (run_every in config.yaml) vs rule time_frame. I am not sure how this works. If say - my monitor frequency is set at run_every 15 minutes, and my rule is looking for 30 count (or more) over time_frame: 5 minutes. Does this scenario work correctly? I was looking at the queries generated, and note that queries are over frequency length (ie 15 minutes). How does the code determine that within a 5 minute interval a 30 count has been reached?
Hi guys. I have a question that I monitor some similar log files in the same time and try to make sure they are continuing seeing heartbeat. So I simply use flatline to do this. There are over 20 log files I need to alter so Is that possible I can put all of the alter into one configure file in a smart way. Otherwise, I will need to run elastalert one for each
hi everyone . I am trying to set an alert on elastalert based on percentage . Currently i am using "type: frequency" but i want it based on percentage. I dont want to give a fix value for alert (eg. "num_events: 50"). is it possible to give "num_events" field as percentage? Please help
@mrdima This has been fixed recently
Flatline alerts now can trigger without any event at all
@YaohuiYu Just add
query_key: filename and the flatline will trigger independently for each value of the field filename
@bob22233 https://www.elastic.co/guide/en/elasticsearch/plugins/current/delete-by-query-usage.html delete where _type: elastalert_status and rule_name: <rulename>
@javatechy Not possible quite yet. This has been a long requested feature though :(
@Qmando I keep getting flatline alerts for my cloudtrail logs (set to fire if no logs are received in 3 hours), even though I can clearly see I'm still getting cloudtrail logs (which should be pretty much continuously checked for). This is the only alert that fires. My cloudtrail collection logstash service runs on the same box as elastalert and kibana. All other logs go through logstash on a different instance. My suspicion is that my alerts are firing because something maybe is being overworked, but I'm not quite sure how to debug that (CPU, memory, and events/second all seem fine). Any ideas on how to debug this?
@bob22233 Several possibilities. The first that comes to mind is that there is some sort of delay when indexing cloudtrail logs. This is especially likely if you are using use_count_query or use_terms_query. For example, at 12:00, elastalert might query for all logs between 11:55 and 12:00. Maybe it gets 0 hits. Some time after that, the cloudtrail log file containing logs for the time period gets processed.
You can confirm this by looking at the logs for elastalert_status. If it queried a time period and got 0 hits, but you see documents during that time period, it's likely they were added with a small delay
You can fix this by adding query_delay
you can also confirm this by looking at Kibana, most recent 15 minutes. What is the most recent log that it shows? I'm guessing it's not RIGHT up to date