Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    he2ss
    @he2ss
    Hi @Lamera,
    you can check your alerts with cscli cscli alerts list or even your crowdsec logs /var/log/crowdsec.log #bydefault and see if there is pull of IPs from CAPI.
    And a question, does the crowdsec instance at work has already installed scenarios ?
    Lamera
    @Lamera
    hi @he2ss
    yes, there are a few collections installed.
    indeed, there is an error:
    time="07-05-2021 13:40:07" level=error msg="capi pull top: get stream: Get https://api.crowdsec.net/v2/decisions/stream?startup=true: could not get jwt token: Post https://api.crowdsec.net/v2/watchers/login: dial tcp 54.194.20.168:443: i/o timeout"
    time="07-05-2021 14:10:07" level=error msg="capi metrics: sending metrics failed: Post https://api.crowdsec.net/v2/metrics/: could not get jwt token: Post https://api.crowdsec.net/v2/watchers/login: dial tcp 34.254.26.228:443: i/o timeout"
    maybe there is something blocked ...
    he2ss
    @he2ss
    oh maybe network issue on your work office, no ?
    weird it's on 443 port
    Lamera
    @Lamera
    i think it's our hosting provider. we use a proxy to reach the internet.
    Sytav
    @sytav_twitter
    Hi. Question. Somebody know how to configure custom scenarios ? (they are very conservative)
    Sytav
    @sytav_twitter
    INFO[07-05-2021 07:42:52 PM] Ignoring file /etc/crowdsec/hub/scenarios/crowdsecurity/ssh-scenario-xxx.yaml of type scenarios
    I can edit existing scenario, but cannot create new one
    registergoofy
    @registergoofy
    hi @sytav_twitter
    try to store them directly in /etc/crowdsec/scenarios
    Sytav
    @sytav_twitter
    FATA[07-05-2021 09:27:31 PM] Failed to sync Hub index with local deployment : failed to scan /etc/crowdsec/hub : unknown configuration type for file '/etc/crowdsec/hub/scenarios/ssh-scenario-xxx.yaml'
    Will try in /etc/crowdsec/scenarios
    Pascal Christen
    @peesc_gitlab

    I've just read the https://crowdsec.net/2021/05/04/multi-server-setup/ article.

    Is it right that we can't group the servers? Like if server-4 and server-5 would be e.g. database server that they only receive from the "database" group and not receiving from the server-2 and server-3?

    Thibault "bui" Koechlin
    @buixor
    Hello @peesc_gitlab ! no you can't do grouping like this within the same API instance for now. However, you can simply use different local API to do the grouping
    Pascal Christen
    @peesc_gitlab
    @buixor hmm but for that we would need a new database for each "group"/local API. I think the current implementation would't scale for us right now.
    Lamera
    @Lamera

    I still need help to make CrowdSec work behind a proxy. I set up the variables $http_proxy and $https_proxy in /etc/profile.d/proxyserver.sh. With the following curl command and the credentials I get an answer:

    curl -XPOST https://api.crowdsec.net/v2/watchers/login --header "Content-Type: application/json" --data '{"machine_id":"","password":"", "scenarios": ["crowdsecurity/ssh-bf"]}'

    Unfortunately CS doesn't recognize that there is a proxy to use and fails to get the list of ips from the CAPI.

    Lamera
    @Lamera
    I solved my issue with the proxy. I had to set the proxy vars in /etc/systemd/system/crowdsec.service.
    Thank you all very much @Crowd-security team! :smile:
    Geoffrey-lvg
    @Geoffrey-lvg

    Hello,

    Following the tutorial to write my own configuration. When I execute the command I get this error,

    ERRO[12-05-2021 18:22:05] Failed to notify(sent: false): <nil>

    Do you know how to help me?

    Thanks

    Thibault "bui" Koechlin
    @buixor
    Hello @Geoffrey-lvg : don't worry about this, it's simply because you're running it in foreground
    (it's about systemd and daemon notifications)
    Geoffrey-lvg
    @Geoffrey-lvg

    Hello,
    Thanks, but i have already a problem.
    When I execute the test command, it does nothing, I have these first warnings at the beginning,

    WARN[0000] can't load CAPI credentials from './config/online_api_credentials.yaml' (missing field) 
    INFO[0000] push and pull to crowdsec API disabled       
    INFO[0000] single file mode : log_media=stdout daemonize=true 
    INFO[13-05-2021 10:36:58] Crowdsec v1.0.13-linux-a19f13ab45a18024ad7ddbf38ef2ff4aadeaaaf5 
    INFO[13-05-2021 10:36:58] Loading grok library /home/debian/Bureau/crowdsec-v1.0.13/tests/config/patterns/ 
    WARN[13-05-2021 10:36:58] prometheus is enabled, but the listen address is empty, using '127.0.0.1' 
    WARN[13-05-2021 10:36:58] prometheus is enabled, but the listen port is empty, using '6060' 
    INFO[13-05-2021 10:36:58] Loading prometheus collectors                
    WARN[13-05-2021 10:36:58] prometheus: listen tcp 127.0.0.1:6060: bind: address already in use

    and this at the end,

    WARN[13-05-2021 10:36:59] Loaded 2 scenarios                           
    INFO[13-05-2021 10:36:59] [file datasource] opening file './x.log'     
    ERRO[13-05-2021 10:36:59] Failed to notify(sent: false): <nil>         
    WARN[13-05-2021 10:36:59] Starting processing data                     
    INFO[13-05-2021 10:36:59] reading ./x.log at once                      
    WARN[13-05-2021 10:36:59] Acquisition is finished, shutting down       
    INFO[13-05-2021 10:36:59] Killing parser routines                      
    INFO[13-05-2021 10:36:59] 127.0.0.1 - [Thu, 13 May 2021 10:36:59 CEST] "POST /v1/watchers/login HTTP/1.1 200 53.001135ms "crowdsec/v1.0.13-linux-a19f13ab45a18024ad7ddbf38ef2ff4aadeaaaf5" " 
    INFO[13-05-2021 10:36:59] 127.0.0.1 - [Thu, 13 May 2021 10:36:59 CEST] "POST /v1/watchers/login HTTP/1.1 200 51.978345ms "crowdsec/v1.0.13-linux-a19f13ab45a18024ad7ddbf38ef2ff4aadeaaaf5" " 
    INFO[13-05-2021 10:37:00] Bucket routine exiting

    I can't understand where this can come from since I followed the tutorial
    Thanks,

    Thibault "bui" Koechlin
    @buixor
    Hey again @Geoffrey-lvg :)
    the warning messages might be confusing (or actually the log level might not be appropriate), but you can safely ignore them

    INFO[13-05-2021 10:36:59] reading ./x.log at once

    your logs gets reads, but no alerts are generating later

    I tink you want to turn debug: on in your parer to see what is going on
    Geoffrey-lvg
    @Geoffrey-lvg

    Hey
    Thanks for you answer
    It's already on on/true

    I use this parser

    filter: 1 == 1
    debug: true
    onsuccess: next_stage
    name: me/myparser
    description: a cool parser for my service
    grok:
    #our grok pattern : capture .*
      pattern: ^%{DATA:some_data}$
    #the field to which we apply the grok pattern : the log message itself
      apply_on: message
    statics:
      - parsed: is_my_service
        value: yes

    And this log

    May 11 16:23:43 sd-126005 kernel: [47615895.771900] IN=enp1s0 OUT= MAC=00:08:a2:0c:1f:12:00:c8:8b:e2:d6:87:08:00 SRC=99.99.99.99 DST=127.0.0.1 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=51006 PROTO=TCP SPT=45225 DPT=8888 WINDOW=1024 RES=0x00 SYN URGP=0 
    May 11 16:23:50 sd-126005 kernel: [47615902.763137] IN=enp1s0 OUT= MAC=00:08:a2:0c:1f:12:00:c8:8b:e2:d6:87:08:00 SRC=44.44.44.44 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=17451 DF PROTO=TCP SPT=53668 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0
    Thibault "bui" Koechlin
    @buixor
    ah, so something is off indeed :)
    when you do cscli hub list what do you see ?
    Geoffrey-lvg
    @Geoffrey-lvg

    Yes, I see this

    -------------------------------------------------------------------------------------------------------------
     NAME                            📦 STATUS   VERSION  LOCAL PATH                                             
    -------------------------------------------------------------------------------------------------------------
     crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml 
     crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml     
     crowdsecurity/sshd-logs         ✔️  enabled  0.5      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml         
     crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml         
     crowdsecurity/whitelists        ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml       
    -------------------------------------------------------------------------------------------------------------
    INFO[13-05-2021 11:15:37 AM] SCENARIOS:                                   
    --------------------------------------------------------------------------------
     NAME                  📦 STATUS   VERSION  LOCAL PATH                          
    --------------------------------------------------------------------------------
     crowdsecurity/ssh-bf  ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml 
    --------------------------------------------------------------------------------
    INFO[13-05-2021 11:15:37 AM] COLLECTIONS:                                 
    --------------------------------------------------------------------------------
     NAME                 📦 STATUS   VERSION  LOCAL PATH                           
    --------------------------------------------------------------------------------
     crowdsecurity/sshd   ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml  
     crowdsecurity/linux  ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml 
    --------------------------------------------------------------------------------
    INFO[13-05-2021 11:15:37 AM] POSTOVERFLOWS:                               
    --------------------------------------
     NAME  📦 STATUS  VERSION  LOCAL PATH 
    --------------------------------------

    But i'm work on the version for tests

    Thibault "bui" Koechlin
    @buixor
    from what I see, the parser you pasted above is not enabled right ?
    (if it was, you would see some debug log from it when it tries to process the existing logs)
    in which tutorial do you rely ?
    Thibault "bui" Koechlin
    @buixor
    ok
    ah, sorry then I meant cscli -c dev.yaml hub list :)
    Geoffrey-lvg
    @Geoffrey-lvg
    Thanks,
    --------------------------------------------------------------------------------------------------------------------------------------------------------
     NAME                            📦 STATUS          VERSION  LOCAL PATH                                                                                 
    --------------------------------------------------------------------------------------------------------------------------------------------------------
     crowdsecurity/dateparse-enrich  ✔️  enabled         0.1      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/parsers/s02-enrich/dateparse-enrich.yaml 
     crowdsecurity/geoip-enrich      ✔️  enabled         0.2      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/parsers/s02-enrich/geoip-enrich.yaml     
     myparser.yaml                   🚫  enabled,local           /home/debian/Bureau/crowdsec-v1.0.13/tests/config/parsers/s01-parser/myparser.yaml         
     crowdsecurity/sshd-logs         ✔️  enabled         0.5      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/parsers/s01-parse/sshd-logs.yaml         
     crowdsecurity/syslog-logs       ✔️  enabled         0.1      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/parsers/s00-raw/syslog-logs.yaml         
    --------------------------------------------------------------------------------------------------------------------------------------------------------
    INFO[13-05-2021 11:19:24 AM] SCENARIOS:                                   
    --------------------------------------------------------------------------------------------------------------------
     NAME                  📦 STATUS   VERSION  LOCAL PATH                                                              
    --------------------------------------------------------------------------------------------------------------------
     crowdsecurity/ssh-bf  ✔️  enabled  0.1      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/scenarios/ssh-bf.yaml 
    --------------------------------------------------------------------------------------------------------------------
    INFO[13-05-2021 11:19:24 AM] COLLECTIONS:                                 
    --------------------------------------------------------------------------------------------------------------------
     NAME                 📦 STATUS   VERSION  LOCAL PATH                                                               
    --------------------------------------------------------------------------------------------------------------------
     crowdsecurity/linux  ✔️  enabled  0.2      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/collections/linux.yaml 
     crowdsecurity/sshd   ✔️  enabled  0.1      /home/debian/Bureau/crowdsec-v1.0.13/tests/config/collections/sshd.yaml  
    --------------------------------------------------------------------------------------------------------------------
    INFO[13-05-2021 11:19:24 AM] POSTOVERFLOWS:                               
    --------------------------------------
     NAME  📦 STATUS  VERSION  LOCAL PATH 
    --------------------------------------
    Thibault "bui" Koechlin
    @buixor
    ah then you have it in the test env
    Geoffrey-lvg
    @Geoffrey-lvg
    yes
    Thibault "bui" Koechlin
    @buixor
    with which command did you launch crowdsec ?
    Geoffrey-lvg
    @Geoffrey-lvg
     ./crowdsec -c ./dev.yaml -file ./x.log -type foobar
    Thibault "bui" Koechlin
    @buixor
    it's kind of weird that you don't see any debug from your parser
    can you try to enable debug in the syslog parser as well ?
    Geoffrey-lvg
    @Geoffrey-lvg
    It's strange
    DEBU[13-05-2021 12:01:32] eval(evt.Line.Labels.type != 'syslog') = TRUE  id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] eval variables:                               id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32]        evt.Line.Labels.type = 'foobar'        id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] + Processing 2 statics                        id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] .Parsed[message] = 'May 11 16:23:43 sd-126005 kernel: [47615895.771900] IN=enp1s0 OUT= MAC=00:08:a2:0c:1f:12:00:c8:8b:e2:d6:87:08:00 SRC=99.99.99.99 DST=127.0.0.1 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=51006 PROTO=TCP SPT=45225 DPT=8888 WINDOW=1024 RES=0x00 SYN URGP=0 '  id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] .Parsed[program] = 'foobar'                   id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] Event leaving node : ok                       id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] move Event from stage s00-raw to s01-parse    id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    WARN[13-05-2021 12:01:32] Acquisition is finished, shutting down       
    DEBU[13-05-2021 12:01:32] eval(evt.Line.Labels.type != 'syslog') = TRUE  id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] eval variables:                               id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32]        evt.Line.Labels.type = 'foobar'        id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] + Processing 2 statics                        id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] .Parsed[message] = 'May 11 16:23:50 sd-126005 kernel: [47615902.763137] IN=enp1s0 OUT= MAC=00:08:a2:0c:1f:12:00:c8:8b:e2:d6:87:08:00 SRC=44.44.44.44 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=17451 DF PROTO=TCP SPT=53668 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0 '  id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] .Parsed[program] = 'foobar'                   id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] Event leaving node : ok                       id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    DEBU[13-05-2021 12:01:32] move Event from stage s00-raw to s01-parse    id=fragrant-haze name=crowdsecurity/non-syslog stage=s00-raw
    Thibault "bui" Koechlin
    @buixor
    seems the events move to s01-parse (the stage where is your custom parser) but somehow never gets to your parser
    can you launch it again with --debug ?
    Thibault "bui" Koechlin
    @buixor
    @Geoffrey-lvg I sent you private message so we can avoid spamming the chan :)
    Geoffrey-lvg
    @Geoffrey-lvg
    yes