Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Epifeny
    @epifeny
    So I tried to provide those that I generated earlier, but curl still fails
    # curl -vk https://localhost:8501/v1/agent/self --cacert /etc/consul/tls/<hidden>-agent-ca.pem --key /etc/consul/tls/<hidden>.pem --cert /etc/consul/tls/<hidden>-key.pem
* About to connect() to localhost port 8501 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8501 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* unable to load client cert: -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
* NSS error -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
* Unknown PKCS #11 error.
* Closing connection 0
curl: (58) unable to load client cert: -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
    Epifeny
    @epifeny
    Sorry, the above last output was using a wrong set of certificates. Nevertheless, I've still got an issue:
    # curl -vk https://localhost:8501/v1/agent/self --cacert ./<hidden>-agent-ca.pem --key ./<hidden>-key.pem --cert ./<hidden>.pem
* About to connect() to localhost port 8501 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8501 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
* NSS error -8178 (SEC_ERROR_BAD_KEY)
* Peer's public key is invalid.
* Closing connection 0
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
    Epifeny
    @epifeny
    I did try to compile curl from source with OpenSSL instead of the CentOS default of NSS and that solved my issue. The question remains, how can I overcome this with a default curl in my CentOS distro that is compiled with NSS and while using the built-in Consul TLS creation tool?
    # ./curl/curl-7.67.0/src/curl -sk https://localhost:8501/v1/status/leader --key ./<hidden>-key.pem --cert ./<hidden>.pem | jq
"<hidden>.<hidden>.<hidden>.<hidden>:8300"
    • I had to <hide> some details for obvious privacy/security reasons.
    Epifeny
    @epifeny
    When generating the server certificates, the tutorial says "servers are provided with a special certificate - one that contains server.dc1.consul in the Common Name". Does server.dc1.consul need to have the server prefix? For ex. server.eu.mydomain.com or can it be hostname.eu.mydomain.com? The tutorial isn't very informative on this matter (IMHO). What's the Consul internal usage for the word server in this case?
    JamieGruener
    @JamieGruener
    @epifeny I don't know what the Consul folks say about this, but when we generate server certificates, we include a laundry list of SANs (subject alternative names) to make sure that the cert includes all of the possible hostnames that we might use to reach the server, including the IP address and the Consul-based FQDN like consul-1.node.consul.
    How do folks handle encrypting intra-cluster communications for database clusters? For example, we're looking to stand up a mongodb replica set and a Cassandra cluster, both of which rely on lots of intra-cluster communications. Do folks ever use sidecar proxies to handle that? Or is it self-signed certs (or certs from a managed CA like Vault)?
    Epifeny
    @epifeny

    @epifeny I don't know what the Consul folks say about this, but when we generate server certificates, we include a laundry list of SANs (subject alternative names) to make sure that the cert includes all of the possible hostnames that we might use to reach the server, including the IP address and the Consul-based FQDN like consul-1.node.consul.

    Yea, I don't think I'm gonna be able to get an answer for this. The docs are lacking detail.

    Jasmine W
    @jnwright
    Hello, all! I work as a product designer on Consul and am looking for research participants for a quick, async study. We want to test a new feature that we're currently designing for a future Consul release. If you're interested in participating, please fill out this short google form: https://forms.gle/QA8zLkRenEqYnyaa9. Thank you!
    chrisvanmeer
    @chrisvanmeer:matrix.org
    [m]
    All filled in
    1 reply
    Psi-Jack
    @psi-jack:matrix.org
    [m]
    Why does consul tls cert create -server -dc dc1 create a cert that has only server.dc1.consul, localhost, and 127.0.0.1? Why specifically server.dc1.consul that it itself does not even seem to resolve?
    Psi-Jack
    @psi-jack:matrix.org
    [m]
    I mean, consul.service.dc1.consul resolves as expected. But where in the heck does server.dc1.consul ever resolve in consul land?
    1 reply
    chrisvanmeer
    @chrisvanmeer:matrix.org
    [m]
    Same goes for the client certificate which has DNS:client.dc1.consul, DNS:localhost, IP Address:127.0.0.1
    Karthick Ramachandran
    @rkarthick
    Question on catalog check (https://www.consul.io/api-docs/catalog#check). Is it expected for the consul server to take the service to "critical" after the timeout "1s" and deregister after "30s" automatically (thereby honoring timeout and DeregisterCriticalServiceAfter), if we dont update the health? In my local runs, I am finding the service to be "passing" even after 30 minutes.
    1 reply
    Blake Covarrubias
    @blake
    I recently wrote a dissector for Wireshark that decodes unencrypted RPC and Gossip traffic from Consul. I figured folks here might have an interest in it. It is still very much a work in progress / proof-of-concept project, but does provide a nice way to view network traffic from Consul. The code can be found here: https://github.com/blake/wireshark-consul-dissector. At the moment, it can only read data from PCAP files. The dissector cannot decode traffic that actively being captured by Wireshark.
    tim
    @timhungdao
    Hello there
    anyone has Out of Memory issue with Consul 1.10.4?
    anytime I start the consul agent (client mode) it eat all the server memory and get killed by the kernel
    Yann Huissoud
    @aiqency
    Similar to this issue: hashicorp/consul#1545 I want to add tags to services programmatically in go. /v1/catalog/register is expecting the full service definition (api.AgentServiceRegistration), but I couldn't find any API to get it in the first place (in order to modify it).
    /v1/catalog/service/ also doesn't return all required values.
    2 replies
    I forgot to specify that this must be done remotely. Basically some code on a machine would query the catalog and update the service tags.
    iluciv
    @iluciv:matrix.org
    [m]
    Hey there I needed to set up dns with iptables forwarding on the mangaement servers following here https://learn.hashicorp.com/tutorials/consul/dns-forwarding?utm_source=consul.io&utm_medium=docs#iptables-setup
    would I need to apply these nat forwarding rules on both the consul agents running in client mode and the consul server cluster agents
    at the moment I only have it set on the management server cluster
    the clients are running as non root and have the port set to 8600 in their config so I'm assuming that they will need the nat forwarding also. Management nodes are in a forward look up zone in our dns.
    iluciv
    @iluciv:matrix.org
    [m]
    oh we have fabio running on the worker nodes (those with consul running in client mode) does that constitute a relay host? (not that flash on dns)
    chrisvanmeer
    @chrisvanmeer:matrix.org
    [m]
    Depends on your DNS setup I guess...in my setup, both Consul servers and clients have dnsmasq installed so that they will default query their own DNS instance. The rest of the servers in a separate VLAN use the standard DNS servers of the environment, which has a conditional DNS forwarder for .consul that forwards to the 5 Consul servers.
    In any way, it would be beneficial when Consul clients would be able to resolve the .consul domain.
    iluciv
    @iluciv:matrix.org
    [m]
    Thanks chrisvanmeer
    Ilya Balashov
    @ibalashov24

    Hello! I have a flood of the following nasty warnings on my Consul installation (v.1.11.3):

    [WARN]  agent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=sth-with_underlines

    Unfortunately, in my case, service renaming is not feasible. At the same time, my setup does not use the DNS interface at all, so complete DNS disabling would be an appropriate solution, I think. I've tried to set a negative DNS port as suggested here: hashicorp/consul#3135 using CLI flag "-dns-port -1" , but it seems to have no effect.
    Could you please advise if there is any way to disable DNS (or solve the warning problem)?

    1 reply
    Narendra Patel
    @narendrapatel
    Hi, we had certificates expiry for 2 of our connect enabled services ( 1 is on VMs and other is on K8s). Consul should have renewed them post the 72 hours window but it didn't.
    We had to restart the service in VM and recreate 1 pod replica for the K8s service to solve the issue. Have avoided restarting the other replica for time being on K8s to further debug the issue.
    4 replies
    Narendra Patel
    @narendrapatel
    One question on monitoring envoy sidecars. Envoy exposes a lot of metrics at the all levels like Listener, Cluster, HTTP level stats, Server level stats, etc.. What should be a good set of key metrics to monitor and perhaps alert on. Istio provides a set of precanned dashboards for use. Do we have some thing similar for Consul?
    1 reply
    Ryan Goltry
    @grogsaxle
    Is there a setting to tell consul to log DNS requests? Not that I'm sold on logging DNS requests on consul, there are other options, but wondering if dns logging in consul is an option.
    Michael Aldridge
    @the-maldridge
    this sounds like an XY problem @grogsaxle what is the problem you're trying to solve?
    5 replies
    Alex Oskotsky
    @aoskotsky-amplify
    Does terminating gateway support connecting to backends that use TLS? I am trying to put an AWS API Gateway behind a Consul terminating gateway
    1 reply
    iluciv
    @iluciv:matrix.org
    [m]
    In a consul cluster should you be setting up the worker nodes to be pointing to the consul server cluster as the dns servers? So for example in address block have the 3 consul server nodes? should the consul server node if using iptables be the only ones with recursers on them or should all node have recursers statement in the config?
    2 replies
    using port 8600 running consul as non root user in this instance
    Mauricio Dantas
    @MauricioIsARed_twitter
    image.png
    1 reply
    did anyone had problems while running consul from systemd?
    Michael Aldridge
    @the-maldridge
    that looks like a pretty standard permissions issue, can you double check that that directory is owned by the expected user?
    ShellFu
    @shellfu

    Hey all, we're running consul on kubernetes. We had to rotate our kubernetes certificates and everything came up fine Consul wise after the restart, however all of the consul-connect-inject sidecars cannot start due to x509 unknown authority "ca".

    I restarted the agents and servers again but this did nothing and am about to attempt the cert rotation process documented.

    Anyone experience this after rotating k8s certs?

    1 reply
    Michael Aldridge
    @the-maldridge
    @blake this page appears to be out of date or otherwise incorrect: https://www.consul.io/docs/connect/configuration it specifies its not required to specify connect { enabled = true } on clients, however without this specified nomad fingerprints the node as attr.consul.connect = false is this a nomad bug or a consul docs bug?
    10 replies
    Daniel Kimsey
    @dekimsey

    I have an ingress-gateway with a service-router to split the L7 traffic (following the docs for HTTP listener with Path-based Routing). But the envoy instance only ever reports "no healthy upstreams".
    Curiously, envoy /clusters shows all the configured upstream clusters (0 on all stats) and /config_dump shows all the routing config looking sane. I'm not 100% clear on what intentions should be set (ingress name -> router or ingress-name -> final destination), but I've currently got a wildcard destination and it's having no effect. And even then I'd expect a 403 response there.
    Logs clearly showing it selecting the configured final-destination cluster (the destination after the service-resolver work is done) and then complaining there are no healthy upstreams. When I look at them in /clusters, I see the correct destination IPs (mesh-gateways) are listed.

    I'm at a loss as to why envoy might be considering the clusters to have no healthy upstreams here.

    3 replies
    Alex Oskotsky
    @aoskotsky-amplify
    Does failover with service-resolver configs support failing over to the closest datacenter if i provide a list of data centers or will it randomly pick from all of them?
    1 reply
    Johan Forssell
    @johanforssell
    intention.png
    This red line - how would I go about listing this from the CLI ?
    I.e. - I would like to know which intentions I've missed without going through the web UI
    consul intention list does not show missing intentions
    iluciv
    @iluciv:matrix.org
    [m]
    I posted this in nomad meant to post it here; Hi there I've inherited some environments, for dns I'm trying to understand why the people before me set the ip in the addresses stanza for dns on the worker nodes (both windows and linux) but on the management nodes they've set is 0.0.0.0. Whats would be the purpose of putting host dns field in there instead of just leaving it as 0.0.0.0 and defaulting to local. Having it this way means for dig queries etc you need to do dig @172.x.x.x -p 8600 some.address ALL instead of just doing dig @localhost
    1 reply
    Michael Aldridge
    @the-maldridge
    if I have a repeated block in my configuration that only changes by label, can I have consul-template iterate over a static list?
    Daniel Kimsey
    @dekimsey
    We are using mesh-gateways to route traffic between our DC's and I'm noticing if there are say 3 instances of a service and 2 are marked as unhealthy (failed_eds_health), the mesh-gateway is still forwarding traffic to them resulting in 1/3 success rate for connections. Even though it's well aware they are offline/dead. Does anyone know if that is an intentional design?
    Daniel Kimsey
    @dekimsey
    Relatedly, what would cause a mesh-gateway endpoint to show up as failed_eds_health in a sidecar's clusters output? For the life of me I cannot figure out what is causing this.