Allright!
For anyone else on the steep learning curve of elastic and security, i had forgotten 2 key parts.
As Antoine accurately pointed out, i had forgotten to sign my CA with the hostnames of my Elastic instances. In my case i added a dns record ie. "host01.somedomain.com" to the list of hosts while running the certificate generator tool listed above, in the step "Enter all hostnames that you need"
I had forgotten to add the certificate .pem file to my metric beat elastic config. Essentially my config ended up along these lines:
output.elasticsearch: # Array of hosts to connect to. hosts: ["https://es01.somedomain.com"] ssl.certificate_authorities: ["/storage/elk-docker/tls/kibana/elasticsearch-ca.pem"] # Protocol - either `http` (default) or `https`. protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "elastic" password: "for_the_love_of_god_change_me_please" # (Not real)
Adding these allowed Metricbeat to authenticate my private CA signed certificate. Im running an ssl config in front of my docker cluster as well using nginx and a reverse proxy, which had its own set of challenges here.
Also please note it's not necessary to regenerate the Certificate Authority in order to generate a new server certificate for ES. If you already have a CA key/cert pair, you can skip directly to the second step of the README.
Yes, however i had not included the proper hostnames in the CA, so i had to redo the CA to include the new host for proper resolution
Just for my own understanding, since you said you have an NGINX instance between your clients and Elasticsearch: are you using Elasticsearch's CA + server certs on NGINX and terminating the TLS connection there? Or are you just doing a passthrough on NGINX and terminating TLS on Elasticsearch directly?
Proxy - I use nginx on my host machine for all dns resolutions. For example i have mapped my elasticsearch to a dns resolvable address on my domain, i.e. 127.0.0.1:9200 to es01.somdomain.com
The reason is 2 fold, i like having all my configuration for (public) port mapping and resolving in one place. The second reason is that i can now just move the pointer to a different cluster should i decide to move my server in the future.
I know that i could do this with an nginx container as well, but i like keeping my webserver seperate from Docker. Makes configuration easier in my workflow at least :-)
command: ['--config.reload.automatic'] somewhere under the logstash service in the Compose file and Logstash will automatically reload your config when it changes.
env:in the Compose file). You should end up with - --config.reload.automatic if you opt for that syntax (dash space dash dash..., looks weird but it's 100% valid).
docker_elk network created by Compose and then you won't even need to expose port 5601 on your host.
command: ['--config.reload.automatic']command:
- --config.reload.automatic
[2021-02-05T15:04:53,567][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>48, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>6000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x18973e34 run>"}and nothing more !
logstash.yml file, contrary to what I told you above: https://www.elastic.co/guide/en/logstash/7.10/logstash-settings-file.html
logstash_1 | [2021-02-05T15:28:21,868][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:main}
logstash_1 | [2021-02-05T15:28:32,879][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
...
logstash_1 | [2021-02-05T15:28:33,592][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x1865a88d run>"}
...
logstash_1 | [2021-02-05T15:28:33,697][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}log.level=debug (can also be set in the config file) and see where it hangs in the logs? (warning: Logstash will produce a LOT of logs in debug mode)
Actually it's a test installation... so there is almost no incoming data (apart from one test server).
At the end it works, but it takes a quite while to restart (around 2mn) as I can see in the logs !
[2021-02-05T15:43:48,802][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:main}
[2021-02-05T15:44:00,577][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-02-05T15:44:04,668][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>48, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>6000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x47483bae run>"}Then it hangs up... but after a while...
[2021-02-05T15:45:39,510][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>94.84}
[2021-02-05T15:45:39,549][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-02-05T15:45:39,553][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}Indeed, brutally restarting the docker container is faster !
elk-stack_logstash latest 278903ffa6ee 3 weeks ago 915MB
elk-stack_kibana latest cca015890e7f 3 weeks ago 1.05GB
docker.elastic.co/kibana/kibana 7.10.2 cca015890e7f 3 weeks ago 1.05GB
docker.elastic.co/elasticsearch/elasticsearch 7.10.2 0b58e1cea500 3 weeks ago 814MB
elk-stack_elasticsearch latest 0b58e1cea500 3 weeks ago 814MB
latest tag are those images).If you know you're not gonna need any plug-in, you can replace
build: directives with image: in the Compose file, and use Elastic's images directly.
