Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    mario96000
    @mario96000
    image.png
    Barry Salas
    @Sixxrett_twitter
    All I have tried to install and reinstall elastalert on one of my elasticsearch servers and everything goes well until I try to create an index. I keep getting elastalert-create-index: command not found.....I have looked at every fix that I could find and ensured that I have all of the needed requirements, however I still cannot get it to create. I am on the latest version of ELK so I am wondering if that could be the issues but I am not sure. Any advice would be greatly appreciated.
    8 replies
    ozgursuder
    @ozgursuder

    Hello there

    I'm generating http post alert with elastalert.

    but the post https I want to write is a site and its certificate is not secure.

    verify_certs: False

    I wonder how can I use the value of elastalert

    alert: post
    http_post_url: "https: // elastic: 9200 / a_alerts / _doc /"

    http_post_headers:
    Content-Type: 'application / json'

    Hernando Agudelo
    @heagma_gitlab

    Any idea why my elastalert make twice a curl call just for 1 alert?

    curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_status/_doc?pretty' -d '{ "@timestamp": "2021-01-12T06:41:26.490784Z", "endtime": "2021-01-12T06:41:26.013716Z", "hits": 0, "matches": 0, "rule_name": "error_filelogs_stage", "starttime": "2021-01-12T06:36:26.013716Z", "time_taken": 0.4770529270172119 }'

    curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_status/_doc?pretty' -d '{ "@timestamp": "2021-01-12T06:41:26.490784Z", "endtime": "2021-01-12T06:41:26.013716Z", "hits": 0, "matches": 0, "rule_name": "error_filelogs_stage", "starttime": "2021-01-12T06:36:26.013716Z", "time_taken": 0.4770529270172119 }'

    PH03NIX
    @tarek_chalan_au_twitter
    @abhinath3399 show me the error if you still have issues with it
    serut
    @serut
    Hello ! I struggle to create a content for rabbitMQ. My message was correctly created into ES, but if I just take that value from ES and put it in a rabbitMQ message, the receiver cannot parse it as it miss some escape characters. I dont see any possibility on the templating, any solution ? Or do I need to retreate the message before posting it to ES to contains enough escape characters to build a JSON two fingers in the nose ?
    alert_text: '{{"logLevel": "{0}", "thread": "{1}", "class":"{2}",  "message": "{3}", "emitter": "{4}", "@timestamp": "{5}", "@log_name": "{6}", "stackTrace": "{7}", "logType": "{8}", "workflow": {9}}}'
    alert_text_type: alert_text_only
    alert_text_args:
      - logLevel
      - thread
      - class
      - message
      - emitter
      - '@timestamp'
      - '@log_name'
      - stackTrace
      - logType
      - workflow
    serut
    @serut
    Oh I think the issue is the triple quoted field on ES. It removes escape characters in what is stored inside ES. To sum up, I cannot send a JSON if the ES stored field is a triple quote, as escape characters won't be there ?
    Barry Salas
    @Sixxrett_twitter
    Okay so I though I could get elastalert to work if I used docker vice just a side by side install with my standing Elasticsearch 7.10 and I also got it to work in my test environment with a full docker environment but that is not the case with my production. In my production I am running CentOs 7 and elasticsearch with version 7.10. Even if I follow step by step using a docker or dockerless install elastalert will not install at all. What I mean by this is I pull the clone from Git
    ensure all requirements are installed including setuptools, run python3 setup.py install, fix the config.yaml file, fix a sudo rule for testing, and go to create index and that is where it stops and I cannot do anything further
    3 replies
    Vlad Vetshtein
    @vvvlad
    aws slack
    wajika
    @wajika

    Hello everyone, I got an error while testing the metric_aggregation rule. Does anyone have an answer?

    2021-01-28 15:40:24,352 INFO elasticsearch GET http://192.168.10.145:9200/elastalert_status_test/_search?size=1 [status:200 request:0.005s]
    2021-01-28 15:40:24,353 ERROR root Traceback (most recent call last):
    File "/usr/local/python36/lib/python3.6/site-packages/elastalert/elastalert.py", line 1269, in handle_rule_execution
    num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
    File "/usr/local/python36/lib/python3.6/site-packages/elastalert/elastalert.py", line 949, in run_rule
    self.add_aggregated_alert(match, rule)
    File "/usr/local/python36/lib/python3.6/site-packages/elastalert/elastalert.py", line 1807, in add_aggregated_alert
    alert_time = ts_now() + rule['aggregation']
    TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'dict'

    My rules file

    name: test
    type: metric_aggregation
    index: apm-*
    timeframe:
    minutes: 1
    metric_agg_key: transaction.duration.us
    metric_agg_type: avg
    query_key: service.name
    sync_bucket_interval: true
    allow_buffer_time_overlap: true
    use_run_every_query_size: true
    bucket_interval:
    seconds: 10
    doc_type: _doc
    max_threshold: 100000
    filter:

    • term:
      agent.name: "dotnet"
      alert:
      • "debug"
    4 replies
    ocelino
    @ocelino
    Hi all, is there a way to send email report every 5 minutes using elastalert and the body is formatted like this:
    API Calls: 2,872
    Resp. Time: 449 ms
    Error Rate: 0.38%
    Ramesh Pradhan
    @rms-sth
    Does latest version of elastalert support kibana_url and use_kibana_dashboard feature for kibana 7.x ?
    7 replies
    Bhanupraveen G
    @bhanupraveeng
    elastalert-test-rule example_rules/example_frequency.yaml by default using " File "/usr/lib/python2.7/site-packages/pkg_resources/init.py"". failing with syntax error. I need to use python3.6 package which is installed. Please let me know?
    1 reply
    Ashish Singh
    @ashishkaransingh

    Hello All,
    I was trying to modify
    alerts.py (/elastalert/alerts.py)
    https://github.com/Yelp/elastalert/blob/1dc4f30f30d39a689f419ce19c7e2e4d67a50be3/elastalert/alerts.py#L1798

    Reason: in order to send events to Servicenow
    To muck around i changed the payload, essentially hardcoded the values except description (and it works it triggers an alert and creates an event in Servicenow).
    https://github.com/Yelp/elastalert/blob/1dc4f30f30d39a689f419ce19c7e2e4d67a50be3/elastalert/alerts.py#L1809
    payload = {
    "records":[{
    "source": "oursource",
    "event_class": "Log_Analytics",
    "node": "comp1",
    "metric_name": "Interface",
    "type": "Details hardcoded",
    "resource": "resource",
    "severity": "3",
    "description": "This is dummy description",
    "additional_info": description
    }]}

    My question is how can i fetch values dynamically from Elasticsearch?
    Example node is a Host.keyword in elasticsearch how can i fetch it?
    I notice description has most of the data already how can make use of it?

    Any help would be great!!

    Thank you

    Ashish Singh
    @ashishkaransingh
    I guess i should ask how can i access the dict object from matches
    https://github.com/Yelp/elastalert/blob/1dc4f30f30d39a689f419ce19c7e2e4d67a50be3/elastalert/alerts.py#L1798
    def alert(self, matches):
    for match in matches:
            # Parse everything into description.
            description = str(BasicMatchString(self.rule, match))
    Ashish Singh
    @ashishkaransingh

    Notice i tried to add "node": match.Host, but got an error

    File \"/opt/elastalert/elastalert/alerts.py\", line 1815, in alert", " \"node\": match.Host,", "AttributeError: 'dict' object has no attribute 'Host'"],

    payload = {
    "records":[{
    "source": "SupportOne",
    "event_class": "Log_Analytics",
    "node": match.Host,
    "metric_name": "Interface",
    "type": "Details hardcoded",
    "resource": "resource",
    "severity": "3",
    "description": "This is dummy description_222",
    "additional_info": description
    }]}

    Ashish Singh
    @ashishkaransingh

    Looks like this is it
    lookup_es_key

    node = lookup_es_key(matches[0], 'Host')
    message = lookup_es_key(matches [0], 'Message')

    then use it in the payload

    beaesteban01
    @beaesteban01

    Hi everyone!
    I am working with a few frequency rules but I am facing the following problem.
    Setting the number of events to 3, as the documentation says "This rule matches when there are at least a certain number of events in a given time frame", if I have more than 3 hits, all of the hits that are in the timeframe should be included in the same alert, not triggering new alerts only each 3 hits, that is what I am getting: matches for only the exact num_events, not more. My rule:

    name: r1_redmov
    description: "3 events (any) redes moviles  in 2 minutes"
    
    type: frequency
    
    # (Required)
    # Index to search, wildcard supported
    index: redes_moviles
    
    # If different than default
    timestamp_field: time_stamp
    num_events: 3
    timeframe:
      minutes: 2
    
    # to see all the hits related to the match
    attach_related: true  
    
    alert:
    - "debug"

    Moreover, looking into the ruletypes.py, in the frequency rule, the condition is if self.occurrences[key].count() >= self.rules['num_events’]:

    Thanks!!

    2 replies
    Parul Kushwaha
    @Parulk
    Hi everyone! I am creating this below elastalert rule :
    rule1.yaml: |-
    # elasticsearch host
    es_host: my_host
    es_port: my_port
    name: Error Log Alerts
    type: frequency
    num_events: 1
    timeframe:
      hours: 1
    
    index: testindex-applicationlogs-*
    
    filter:
    - query:
        query_string:
          query: "appname:xxx AND host_stability:prod AND (function:web.RemotingServlet.handleThrowable* OR loglevel:ERROR)"
    
    
    alert:
    - "slack"
    alert_subject: "Error Log Alerts"
    slack:
    slack_webhook_url: "slack_webhook_url"
    I am getting this exception while running the rule : ERROR:root:Uncaught exception running rule Error Log Alerts: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [<http_request>] would be [31155871676/29gb], which is larger than the limit of [31045425561/28.9gb], real usage: [31155870848/29gb], new bytes reserved: [828/828b], usages [request=0/0b, fielddata=60671194/57.8mb, in_flight_requests=9705526/9.2mb, accounting=223410932/213mb]')
    Wanted to understand how to introduce the timestamp in filter ?
    Something like to query for last 1 hour.
    ShotgunSpider
    @ShotgunSpider
    if i want to get a 40% drop in events using spike_alert should the spike_height be 1.6 for 40% and spike_type be down?
    Ashish Singh
    @ashishkaransingh

    Hi,

    Need help in accessing alert or rule types used in .yaml
    /elastralert/elastalert/alerts.py
    I am able to access and print values filter and name.

    image.png

    How can i access alert type example whether it is frequency or Average etc
    I tried as get_rules = self.rule.get('type', None) but it print the following line including the class etc
    "Print rules: <elastalert.ruletypes.FrequencyRule object at 0x7fd3e204c310>"

    A little background:
    I am creating a custom payload for class ServiceNowAlerter(Alerter): in alerts.py

    RashmiChoudhary
    @rashmichoudhary07
    Hello @Qmando Hope you are doing well...
    I have an issue in production environment(at work) where elastalert is showing a high time_taken values in elastalert_status_status index. This happens intermittenly for a continuous timeframe during certain times in a day which is troubling my client since eventually real time alerts are delayed. I have been trying to find Root cause for the delay introduced at elastalert but no luck since it is intermittent. Please find a sample document from prod for your help.@timestamp: 2021-02-18T07:45:16.048284Z
    _id: SZkZtHcBUW6Vb-ZGp3YW
    _index: elastalert_status_status
    _type: _doc
    endtime: 2021-02-18T06:03:42.463717Z
    hits: 407
    matches: 34
    num_hits: 3
    num_matches: 1
    rule_name: Shop - Severity - Minor [400 TO 499] And Not [401,403,424]
    starttime: 2021-02-18T05:38:43.224775Z
    time_taken: 6093.584537982941Is time_taken the exact time for elastalert to only query elasticsearch?
    We have millions of documents in our ES and Kibana gives faster result for the same query for time_taken is a high value. Also, if i manually run the cURL query that elastalert generates the result is fetched only with in few 100 milliseconds. We are seeing 1 to 4+ hours to time_taken value in producton and unable to get to the root cause of this.Could please help us to get to the root of high time_taken value. I am stuck in a real prod issue here with elastalert delaying alert triggers due to huge time_taken value(in seconds as per doc). Looking forward to hearing from you.
    I have trying to research on what is causing the above but no luck yet
    i have also opened ticket at github issues site
    RashmiChoudhary
    @rashmichoudhary07
    i am new to this platform Any help from experts is highly appreciated.. TIA!
    Ashish Singh
    @ashishkaransingh

    Hello,
    Anybody running Elastalert using python 3.8
    I am getting error

    python3 -m elastalert.elastalert --verbose --rule test_count_alert.yaml --config config.yaml

    Throwing error
    
    Traceback (most recent call last):
      File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
        mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
      File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
        __import__(pkg_name)
     File "/opt/elastalert/elastalert/elastalert.py", line 29, in <module>
        from . import kibana
    ImportError: attempted relative import with no known parent package

    Ref
    https://github.com/Yelp/elastalert/issues/2439#issuecomment-534058583
    Yelp/elastalert#2776

    Ashish Singh
    @ashishkaransingh
    I figured it out...
    Kumar ankit
    @Ankitxroot

    pkg_resources.DistributionNotFound: The 'elastalert==0.1.28' distribution was not found and is required by the application

    How to fix this.
    Thanks in advance.

    4 replies
    danny zak
    @dannyzak_gitlab
    is there a way to specify a timewindow when alerting should (not) occure for a certrain rule, eg. not after friday 5pm till monday 9am
    Kyle Parrish
    @arnydo

    Hiya! Anyone have any feedback on building filters? I am trying to create an alert for Office 365 sign ins outside of the US. I am trying thie:

    filter:
    - query:
            query_string: {query: "NOT source.geo.country_iso_code:US"}
            query_string: {query: 'o365.audit.Operation: UserLoggedIn'}

    But...it doesn't seem to work as expected. I can't find documentation on building boolean filters. "this AND NOT that..."

    7 replies
    Kyle Parrish
    @arnydo
    Also, why are there hits when running elastalert-test-rule but not elastalert? Both are using the same rule.
    ankita1596
    @ankita1596
    My elastalert and elasticsearch is running in docker environment. Can you please help me to send output from elastalert to zabbix running in docker.
    dunter-d
    @dunter-d
    hi! I'm having issues with elastalert. Running ubuntu 20.04/elasticsearch 7.10/python3.8 & running systemctl start elastalert outputs this:
    '''
    rule['alert'] = self.load_alerts(rule, alert_field=rule['alert'])
    File "/usr/local/lib/python3.8/dist-packages/elastalert-0.2.4-py3.8.egg/elastalert/loaders.py", line 469, in load_alerts
    alert_field = [normalize_config(x) for x in alert_field]
    File "/usr/local/lib/python3.8/dist-packages/elastalert-0.2.4-py3.8.egg/elastalert/loaders.py", line 469, in <listcomp>
    alert_field = [normalize_config(x) for x in alert_field]
    File "/usr/local/lib/python3.8/dist-packages/elastalert-0.2.4-py3.8.egg/elastalert/loaders.py", line 450, in normalize_config
    config_copy.update(config) # warning, this (intentionally) mutates the rule dict
    ValueError: dictionary update sequence element #0 has length 1; 2 is required
    elastalert.service: Main process exited, code=exited, status=1/FAILURE
    '''
    dunter-d
    @dunter-d
    @ashishkaransingh how did you solved that issue with the "from . import kibana" error?
    dplgrail
    @dplgrail
    image.png
    Hi, i've this issue... TestController: Failed to test rule with error: INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
    To send them but remain verbose, use --verbose instead.
    Im using SIEMonster and Praeco for ElastAlert
    1 reply
    raks25
    @raks25:matrix.org
    [m]
    Hi can anyone help me out with this "1|INFO|0|1|ibs| |2021/04/22 18:54:30.999234|http: TLS handshake error from **: remote error: tls: bad certificate" log displayed on my kibana and i am using elastalert for notifying the string " tls: bad certificate" but it sends alerts when we have word like only certificate in it , but i want to be notified only when the log has keyword "bad certificate"
    Naoyuki Sano
    @nsano-rururu
    Yelp/elastalert is no longer maintained. Please use jertel/elastalert2. Questions to the discussion below
    https://github.com/jertel/elastalert2/discussions
    itruivitorino
    @itruivitorino
    Hi everyone. I've been trying to make an alert go when the status of a heartbeat monitor goes up AFTER being down. Anyone made any progress with similar thing?
    ramprasadavirineni
    @ramprasadavirineni:matrix.org
    [m]

    Hi @nsano-rururu
    Hello all,
    I've been installed elastalert on elk7.11, I configured config and rules files, but alert are not getting on mails..
    please anyone have idea on help me out....

    1 rules loaded
    INFO:elastalert:Starting up
    INFO:elastalert:Disabled rules are: []
    INFO:elastalert:Sleeping for 59.999878 seconds
    INFO:elastalert:Queried rule Hello Test mail from ELK Stack please ignore from 2021-05-07 07:41 EDT to 2021-05-07 07:56 EDT: 0 / 0 hits
    INFO:elastalert:Ran Hello Test mail from ELK Stack please ignore from 2021-05-07 07:41 EDT to 2021-05-07 07:56 EDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
    INFO:elastalert:Background configuration change check run at 2021-05-07 07:57 EDT

    INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-05-07 07:57 EDT

    My config.yml:
    cat config.yaml|grep -v "^#"
    rules_folder: example_rules

    run_every:
    minutes: 1

    buffer_time:
    minutes: 15

    es_host: 192.168.0.1

    es_port: 9200

    writeback_index: elastalert_status
    writeback_alias: elastalert_alerts

    alert_time_limit:
    days: 0

    smtp_port: 25

    smtp_host: 'xxx.com'

    name: Hello Test mail from ELK Stack please ignore

    type: frequency

    index: filebeat-*

    num_events: 3

    timeframe:
    hours: 1

    filter:

    • query:
      query_string:
      query: "message: authentication failure OR failure password"
      timestamp_field: "@timestamp"

      - term:

      process_name: "DefaultQuartzScheduler6"

    alert:

    • "email"

    email:

    Naoyuki Sano
    @nsano-rururu
    @ramprasadavirineni:matrix.org
    I don't use elastalert.
    ramprasadavirineni
    @ramprasadavirineni:matrix.org
    [m]

    Hello,

    could you please suggest which one good for alert notification for ELK 7.x