Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Quentin Long
    @Qmando
    using pipe_match_json
    Abhishek Garai
    @abhigarai
    go
    Thanks for the info @Qmando I'll get going with the stuff..
    Ying
    @ying1
    @Qmando thank you! Is there an issue that I can subscribe to ?
    Ying
    @ying1
    Hello - I am looking at making lots of rules that looks similar (for example - check number of RC 401 for a set of end points). Are there any suggestions as to what is the best way to do this ? Should it be one rule per end point, or one rule for all end points? I am assuming that multiple rules = multiple queries... is there a way to consolidate the results and parse ?
    Quentin Long
    @Qmando
    @ying1 I guess Yelp/elastalert#169 is the issue to watch. Expect it in the new few weeks
    @ying1 You can do all end points in a single rule
    There are several features that support this. You can use query_key which will group documents by certain key and then count them separetely. So for example, if you have frequency type with num_events: 10 plus query_key: hostname, it would only alert if a single hostname got more than 10 events
    Query_key also works for other types, basically just applies the log separately for each unique value of that field (or combo of yields)
    There's also use_terms_query which will make a terms aggregation query instead of normal query if there are too many documents
    Ying
    @ying1
    Thanks!!!!!!!
    mchesmo3
    @mchesmo3
    I am having a devil of a time getting elastalert installed on Centos 7. I think I have dependency issues...
    Quentin Long
    @Qmando
    Did you pip install or git clone?
    You should use a virtualenv if you are not already
    pip install elastalert will try to install all of the latest version of each dependency. If you pip install -r requirements.txt, it will use a known working (and compatible) list dependencies, though some of those versions are pretty old
    If you are getting compile errors for blist, that's indicative of a messed up pip installation
    mchesmo3
    @mchesmo3
    I think it is the pip installation. I am going to work on it today...tx
    mchesmo3
    @mchesmo3
    I used the git clone. When you say virtualenv do you mean a Docker instance or a whole other Virtualbox?
    mchesmo3
    @mchesmo3
    I think it is a compile issue. If its ok I am going to post the error.
    Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-5RHl3s/blist/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-6lrisv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-5RHl3s/blist/
    pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)
    version ^
    I get the above error when I try and run pip install elastalert as root
    mchesmo3
    @mchesmo3

    When I run python setup.py install I get the following error Running blist-1.3.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xDDIzl/blist-1.3.6/egg-dist-tmp-PPWuCk
    warning: no files found matching 'blist.rst'
    blist/_blist.c:38:20: fatal error: Python.h: No such file or directory

    include <Python.h>

                    ^

    compilation terminated.
    error: Setup script exited with error: command 'gcc' failed with exit status 1
    I think this a PiP issue but I dont know how to fix it yet.

    mchesmo3
    @mchesmo3
    I think I figured it out. I needed python-dev installed as well
    mchesmo3
    @mchesmo3
    @mchesmo3
    mchesmo3
    @mchesmo3
    I thought I had it fixed but when I try and run the elastalert=create index I get a huge error... pretty much the same think as when I try and test a rule using elastalert-test
    mchesmo3
    @mchesmo3
    Traceback (most recent call last):
    File "/bin/elastalert-create-index", line 11, in <module>
    load_entry_point('elastalert==0.1.2', 'console_scripts', 'elastalert-create-index')()
    File "/usr/lib/python2.7/site-packages/elastalert-0.1.2-py2.7.egg/elastalert/create_index.py", line 116, in main
    if es_index.exists(index):
    File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(args, params=params, *kwargs)
    File "/usr/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 212, in exists
    self.transport.perform_request('HEAD', _make_path(index), params=params)
    File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
    File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 67, in perform_request
    raise ConnectionError('N/A', str(e), e)
    elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ConnectionError(('Connection aborted.', error(111, 'Connection refused')))
    bob22233
    @bob22233
    How can I make elastalert forget it ever ran a rule? One option is to change the rule name, but I assume there is something in the elasticsearch index /elastalert_status/ that I can delete in order to make it forget.
    mchesmo3
    @mchesmo3
    This may such a basic question that it seems silly. Should I run elastalert on the same server as Elasticsearch?
    mchesmo3
    @mchesmo3
    When I point the ES_host can I do it by IP address? When I set the es_port does it matter what port I set it to?
    mrdima
    @mrdima
    @Qmando Coming back to my question of Sep 8 :smile: If I want to alert when there are no matches, the flatline only works if there was a match first and then drops to 0 in for example 24 hours. I want to alert with 0 matches. Custom ruletypes/alerters/enhancements won't work as they are all seem to be only triggered when matches occur. Is there any way to specify and alert when there are no matches? (example usecase "virusdefinitions not updated", it's easy to find out with 1 query if some line about updating occurs...if it's 0 times in the last 24 hours and alert has to be sent...we cannot expect that there ever was a succesful line about updating beforehand)
    Ying
    @ying1
    Hello - have a question relating to monitor frequency (run_every in config.yaml) vs rule time_frame. I am not sure how this works. If say - my monitor frequency is set at run_every 15 minutes, and my rule is looking for 30 count (or more) over time_frame: 5 minutes. Does this scenario work correctly? I was looking at the queries generated, and note that queries are over frequency length (ie 15 minutes). How does the code determine that within a 5 minute interval a 30 count has been reached?
    YaohuiYu
    @YaohuiYu
    @mrdima I have the same problem with you. I use logstash to send an heartbeat message and make sure my query can find the heartbeat message each time so flatline rule always has at least 1 match. but not sure if there is some easier way
    YaohuiYu
    @YaohuiYu
    Hi guys. I have a question that I monitor some similar log files in the same time and try to make sure they are continuing seeing heartbeat. So I simply use flatline to do this. There are over 20 log files I need to alter so Is that possible I can put all of the alter into one configure file in a smart way. Otherwise, I will need to run elastalert one for each
    javatechy
    @javatechy
    hi everyone . I am trying to set an alert on elastalert based on percentage . Currently i am using "type: frequency" but i want it based on percentage. I dont want to give a fix value for alert (eg. "num_events: 50"). is it possible to give "num_events" field as percentage? Please help
    Marek Hobler
    @neutrinus
    Hi elastalert!, is there any way to trigger an alert when the frequency is below certain number? I would like to check if the application is logging properly
    Quentin Long
    @Qmando
    @neutrinus type: flatline
    @mrdima This has been fixed recently
    Flatline alerts now can trigger without any event at all
    @YaohuiYu Just add query_key: filename and the flatline will trigger independently for each value of the field filename
    @bob22233 https://www.elastic.co/guide/en/elasticsearch/plugins/current/delete-by-query-usage.html delete where _type: elastalert_status and rule_name: <rulename>
    @javatechy Not possible quite yet. This has been a long requested feature though :(
    bob22233
    @bob22233
    @Qmando I keep getting flatline alerts for my cloudtrail logs (set to fire if no logs are received in 3 hours), even though I can clearly see I'm still getting cloudtrail logs (which should be pretty much continuously checked for). This is the only alert that fires. My cloudtrail collection logstash service runs on the same box as elastalert and kibana. All other logs go through logstash on a different instance. My suspicion is that my alerts are firing because something maybe is being overworked, but I'm not quite sure how to debug that (CPU, memory, and events/second all seem fine). Any ideas on how to debug this?
    Quentin Long
    @Qmando
    @bob22233 Several possibilities. The first that comes to mind is that there is some sort of delay when indexing cloudtrail logs. This is especially likely if you are using use_count_query or use_terms_query. For example, at 12:00, elastalert might query for all logs between 11:55 and 12:00. Maybe it gets 0 hits. Some time after that, the cloudtrail log file containing logs for the time period gets processed.
    You can confirm this by looking at the logs for elastalert_status. If it queried a time period and got 0 hits, but you see documents during that time period, it's likely they were added with a small delay
    You can fix this by adding query_delay
    you can also confirm this by looking at Kibana, most recent 15 minutes. What is the most recent log that it shows? I'm guessing it's not RIGHT up to date
    bob22233
    @bob22233
    @Qmando Thanks!