For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.
Hello Ladies & Gentlemen, I've installed Consul as tls enabled on Kubernetes. I am not able to find the same certificate in any secret which is shown here
kubectl exec consul-consul-server-0 -n consul -- curl -sk https://localhost:8501/v1/connect/ca/roots | jq -r .Roots[0].RootCert Could you please advise?
Hi, needed a bit of clarification around the below,
The service name registered in Consul will be set to the name of the Kubernetes service associated with the Pod. This can be customized with the consul.hashicorp.com/connect-service annotation. If using ACLs, this name must be the same as the Pod's ServiceAccount name.
Is it mandatory for the service name to be same as service account or is it only mandatory when we use the annotation consul.hashicorp.com/connect-service?
2 replies
In our case we have an app deployed on k8s which consists of multiple pods, all of which use the same service account. How would consul name services in this case? Also since we annotate at pod spec level how does consul identify its associated service since as per docs the consul service name is set to the name of the Kubernetes service associated with the Pod.
1 reply
Hi, Is there a way to disable circuit breaking for the local app cluster. Facing some very high latency issues when scaling number of requests.
local_app::default_priority::max_pending_requests::1024
local_app::default_priority::max_requests::1024
local_app::default_priority::max_retries::3
local_app::high_priority::max_connections::1024
local_app::high_priority::max_pending_requests::1024
local_app::high_priority::max_requests::1024
local_app::high_priority::max_retries::3
Hey there, I'm hoping someone can clarify where I'm failing here. I simply want to create two tokens so 2 sets of dev teams have access to 2 sets of key values for their apps. We have for instance app-kv key store and 2 folders in that app-1 app-2 I have two policies (policy-dev-team1 and policy-dev-team2) that are simply key_prefix "app-kv/app-1"{ policy="write"} and key_prefix "app-kv/app-2" { policy = "write"} and I've generated two tokens token-dev-team1 and token-dev-team2 attaching to these policies respectively. This approach works with OSS consul v1.9.3 but with the ent v 1.10.4 each token can see the app-kv store but nothing under that. If I change the policy to key_prefix "app-kv" {policy="write"} the tokens can see everything underneath (so both app-1 and app-2 kv's) I know it's not set up correctly how it is (very new to this and this is an inherited setup.) so happy to be schooled on how it should be laid out (for instance should I just create a dev-team-1 and dev-team-2 namespace? I was just looking for a quick segregation of access without having to refactor how these key values are stored/called. Thanks for taking the time to read the ramble
2 replies
Hi, been going through https://learn.hashicorp.com/tutorials/consul/docker-compose-auto-config, is it somehow possible to avoid using hostname in `vault write identity/oidc/ .... template='{"consul": {"hostname": "consul-client" } }'? something like one of node_meta keys or even a wildcard?
This seems to only happen in consul 1.9 and above
I see in the logs that the "<service name>-sidecar-proxy:1" check is getting synced every few minutes. It seems to get reset into critical state until the check is completed. This didn't happen in earlier versions of consul. Any idea what would cause it to repeatedly sync that check when nothing has changed about it?
Thanks @blake will read through that. We really need to plan namespaces more thoughtfully I was really just wanting a quick and dirty policy to get the devs going then refactor with a better structure
Any one know of a chat to talk about the fabio load balancer at all?
1 reply
1.11.2 deb package does not ship with empty /etc/consul.d/consul.env anymore which is used by consul.service:
[Service]
EnvironmentFile=/etc/consul.d/consul.env
...this causes consul to fail on startup
Jan 14 10:47:39 REDACTED systemd[1]: consul.service: Failed to load environment files: No such file or directory
Hey @blake thanks just wanting more to understand how it's working. We recently spun up a dev ent version hashistack consul, nomad,vault and fabio for nomad conatiner lb /ssl I had made a full chain pem file for the wildcard cert to be used by fabio for the nomad jobs. Fabio is configured as a nomad job (docker conainters) on the linux worker nodes and as a windows service on the windows worker nodes. As the linux fabio are docker containers deployed by nomad they check into consul with a 443 health check and the fabio url webprefix. The windows services have been set up as a script to install the service, the fabio windows service config (fabio.properties file and certs) is on a cifs share and it only checks into consul with an 8080 healthcheck. We had an issue with any windows containers were not getting issued ssl certs for their webprefix url until I added in a key.pem and cert.pem file with the full chain pem file I already had in the certs folder. So that issue got resolved but I don't fully understand whats happening here. As the linux jobs are the only ones with the web prefix and the 443 health check in consul are they the only nodes issuing the certs? Is there something I should have configured to improve this? I really had a hard time finding logs of any vaule on the windows nodes. Should I be setting the cert path to consul or vault instead? At the moment the nomd jobs mounts a volume (nfs) and ingests the certs from there and the windows as I said gets they're config and certs off a cifs share. (fwiw we have 4 windows worker nodes and 4 linux worker nodes currently in this cluster)
I have:
- 3 instances with public ips, labelled as separate datacenters.
- firewall rules to allow tcp/udp on 8300-8302 ports.
- bind_addr on all 3 nodes as 0.0.0.0
- network rules are fine since I can ssh to all these nodes from each other over their public ips.
yet I keep getting no route to host while trying to WAN join these togather, any ideas why?
Hi all,
I got this error,
Error creating: Internal error occurred: failed calling webhook "consul-consul-connect-injector.consul.hashicorp.com": failed to call webhook: Post "https://consul-consul-connect-injector-svc.myconsulns.svc:443/mutate?timeout=10s": dial tcp 10.105.230.122:443: i/o timeoutwhile I was trying to deploy the Deployment.
My 3 Consul servers are outside Kubernetes.
I install Consul clients using Helm.
(All Consul v1.11.2)
when I use this command to see logs
"kubectl -n myconsulns logs -f pod/consul-consul-connect-injector-webhook-deployment-59d4d85bg79sk"
2022-01-18T05:58:32.603Z ERROR controller.endpoints failed to get service instances {"name": "consul-consul-connect-injector-svc", "error": "Get \"http://192.168.90.187:8500/v1/agent/services?filter=Meta%5B%22k8s-service-name%22%5D+%3D%3D+%22consul-consul-connect-injector-svc%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22myconsulns%22+and+Meta%5B%22managed-by%22%5D+%3D%3D+%22consul-k8s-endpoints-controller%22\": dial tcp 192.168.90.187:8500: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).ReconcileIn Helm, if I set "connectInject.failurePolicy " to be "Fail", I got the above error.
(If I set "connectInject.failurePolicy " to be "Ignore", I will not get the above error. But there are no SlideCars when I see via "consul catalog services" command)
Could anyone suggest to me how to solve this issue, please.
@blake I just finished reading the roblox postmortem. My hat is off to the consul team for really stepping up and debugging that close to the metal. Going inside the storage primitives really demonstrates how dedicated hashicorp is to building a quality product. Figured you might be able to pass along the positive feedback!
1 reply
@blake Can you please look into this when you got a minute. Feeling stuck as this is a critical use case for us to recommend building a service mesh using consul enterprise.
https://discuss.hashicorp.com/t/setup-envoy-as-ingress-gateway/34547/2
https://discuss.hashicorp.com/t/setup-envoy-as-ingress-gateway/34547/2
1 reply
Hey, I'm trying to connect two services using Consul Connect, I'm doing a sidecar via Nomad and I'm constantly getting "gRPC config stream closed since connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED". I have TLS enabled on Consul and Nomad, using self-signed certificates. Can someone help me with that? I'm not sure if I'm missing some step from the docs, but I thought Consul would handle the
Sorry, the message went cut in half, but to complete: I thought Consul would handle the connection between everyone with its built-in CA. That's it :)
I am using the
consul-k8s project and using it to sync k8s to consul. Looks like that services are synced to consul only on what is called fullSync and on changes. From the comments in the code of the consul-k8s it seems that it should sync changes when they happen and only on full sync (happens every 30 seconds by default). Have anyone else encountered this? is this on purpose?
(Better jargon: a 'follower' server)
Hello all, I am running a 3 node consul cluster (version v1.11.2), when i perform a DNS look up from consul cluster node 01,:
dig +short @localhost -p 8600 consul.service.us-west.consul SRV +tcp
1 1 8300 <REDACTED>-consul-server-01.node.us-west.consul.
1 1 8300 <REDACTED>-consul-server-02.node.us-west.consul.
1 1 8300 <REDACTED>-consul-server-03.node.us-west.consul.However from consul cluster node 02 and 03, I don't get an authoritative answer for
dig +short @localhost -p 8600 consul.service.us-west.consul SRV +tcpAny pointers to debug this is super appreciated
3 replies
blake: But, I think you answered my question. :)
Or match a token with an AccessorID?
We have a setup with an ingress gateway in one datacenter and services that it routes to in other datacenters (using a failover block of a service-resolver) but it doesn't reflect in the topology view (running 1.9.7)
Is this something that is reflected in the topology in a newer version of consul? I wanted to ask before I submitted a feature request that has already been implemented
I submitted a feature request. I actually think this might be a bug because they are listed in the upstreams tab, just not topology hashicorp/consul#12353
Has Consul been removed from the service discovery section?
My bad
Hey folks! I have a 1 node consul cluster that I'd like to upgrade to a 3 node cluster. I will add 2 new nodes, and then once they join the cluster successfully I will then replace the initial node with a new 3rd node. I can't find any documentation or conversation around upgrading from 1 node to 3 node cluster, so just wondering if y'all had any callouts or pitfalls I might hit.
1 reply
Hello, I've got Consul 1.9.7 installed with TLS. I used the built-in CA of Consul to generate the server and client certificates. I followed the Secure Consul Agent Communication with TLS Encryption tutorial. I now want to use the API and am getting some errors from curl:
# curl -k https://localhost:8501/v1/agent/self
curl: (58) NSS: client certificate not found (nickname not specified)What might be the issue here?
I do have
"verify_incoming": true, in the configuration file. So from what I understand, I need to supply the relevant certificates with curl
So I tried to provide those that I generated earlier, but
curl still fails# curl -vk https://localhost:8501/v1/agent/self --cacert /etc/consul/tls/<hidden>-agent-ca.pem --key /etc/consul/tls/<hidden>.pem --cert /etc/consul/tls/<hidden>-key.pem
* About to connect() to localhost port 8501 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8501 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* unable to load client cert: -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
* NSS error -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
* Unknown PKCS #11 error.
* Closing connection 0
curl: (58) unable to load client cert: -8018 (SEC_ERROR_UNKNOWN_PKCS11_ERROR)
Sorry, the above last output was using a wrong set of certificates. Nevertheless, I've still got an issue:
# curl -vk https://localhost:8501/v1/agent/self --cacert ./<hidden>-agent-ca.pem --key ./<hidden>-key.pem --cert ./<hidden>.pem
* About to connect() to localhost port 8501 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8501 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
* NSS error -8178 (SEC_ERROR_BAD_KEY)
* Peer's public key is invalid.
* Closing connection 0
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)
I did try to compile
curl from source with OpenSSL instead of the CentOS default of NSS and that solved my issue. The question remains, how can I overcome this with a default curl in my CentOS distro that is compiled with NSS and while using the built-in Consul TLS creation tool?# ./curl/curl-7.67.0/src/curl -sk https://localhost:8501/v1/status/leader --key ./<hidden>-key.pem --cert ./<hidden>.pem | jq
"<hidden>.<hidden>.<hidden>.<hidden>:8300"- I had to <hide> some details for obvious privacy/security reasons.
When generating the server certificates, the tutorial says "servers are provided with a special certificate - one that contains server.dc1.consul in the Common Name". Does
server.dc1.consul need to have the server prefix? For ex. server.eu.mydomain.com or can it be hostname.eu.mydomain.com? The tutorial isn't very informative on this matter (IMHO). What's the Consul internal usage for the word server in this case?
@epifeny I don't know what the Consul folks say about this, but when we generate server certificates, we include a laundry list of SANs (subject alternative names) to make sure that the cert includes all of the possible hostnames that we might use to reach the server, including the IP address and the Consul-based FQDN like consul-1.node.consul.