For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.
i had some issues after a restart. it was in single cluster node and i tried adding more nodes as it couldnt find leader
i removed the extra nodes again but it still tryes to connect to them. i tried force removing them and everything. but it still comes up in the log that it is trying to connect to it
I cant find anywhere that the old nodes exists, consul members or raft peers dosnt list them
We're currently testing out consul to be used a load balancer type setup via dns. I've been stress testing our test nodes with dns queries. These are bare metal nodes with 32 cpu cores, 128gb of ram, 10gig networking. The best I can seem to get out of consul is 32,000 queries per second, which is basically 1000 queries per second per cpu core. While I'm hitting it with the stress test it's load average, cpu usage, memory usage, and network usage all remain relatively low. I have caching enabled within consul set at 10 seconds. Does anyone have any idea for other config options I could try to squeeze more performance out of this, or am I just hitting some kind of programmatical limit within consul?
I would though if I were inclined to do that put unbound in front of consul and use caching
but cool, I'll look in to unbound, thanks
can anyone point me at the documentation that lists the parameter to change the consul hosted domain name from .consul to something else, I know it exists as I’ve read it and used it before, but I cannot find it now for the life of me, also is there any negative impact from changing this, I can’t think of one, but be nice to know if there is anything I’ve not considered
Hi everyone, I've started a 4 node cluster, this is my config https://gist.github.com/0xalex88/a609af0698d0dd46b48571ea6baed42b I've followed https://developer.hashicorp.com/consul/tutorials/production-deploy/deployment-guide
however the nodes are all continuously logging that there is no leader, what should I do? is this expected?
however the nodes are all continuously logging that there is no leader, what should I do? is this expected?
@0xalex88 Maybe you are just experimenting, but a 4 node cluster is not a good idea.
Consul does leadership election based on majority. In a 4 node cluster, you are likely to run into a split brain or inability to elect a leader.
I believe they are adding a warning for when people incorrectly set an even number in bootstrap_expect.
You want bootstrap_expect set to an odd number:
1 - No HA.
3 - Tolerance for 1 node failing
5 - Tolerance for 2 nodes failing
1 reply
hi there, I'm trying to setup consul inject. I've deployed my 2 pods, I've added the
consul.hashicorp.com/connect-service-upstreams: foo:1234 annotation to the client, but curl localhost:1234 fails with Connection refused and indeed there's nothing listening on that port. curl foo is working. The envoy-sidecar is injected. What could be missing?
one thing I don't understand in the example is why you should define the port in the service client: https://developer.hashicorp.com/consul/docs/k8s/connect#connecting-to-connect-enabled-services
Hi. I'm facing a weird situation between Vault and Consul. Maybe someone here can help me. I have a 5-node Consult cluster and a 5-node Vault cluster, both using latest versions. This uses 5 machines only, each machine holds a member for each service cluster. Vault reports directly to the local Consul server agent. These 5 machines span 3 "geographic/network zones". One zone contains only one node. There was an issue with one of the zones, so two nodes were isolated from the other 3, but that was temporary. The problem I'm seeing now is that although there is only one active/leader Vault node, Consul DNS and service check metric insist to report that two Vault nodes are active, which is not true. For example, DNS querying active.vault.service.mydc.consul alternates between two Vault nodes, and the service check metrics collected from Consul also report those two same nodes. I have no idea what's going on here. Any idea? TIA.
Hi everyone, after following https://developer.hashicorp.com/consul/tutorials/get-started-vms/virtual-machine-gs-deploy#create-server-tokens I'm still getting:
agent: Node info update blocked by ACLs: node=7f08f176-a3f3-effe-7443-bd60865e09d1 accessorID=e340e34c-4ef6-5adb-ad48-5a3d923355f9
agent: Coordinate update blocked by ACLs: accessorID=e340e34c-4ef6-5adb-ad48-5a3d923355f9what could be the reason?
2 replies
the accessor has an ID so I guess a token is set?
the accessor ID matches the "server agent token" that has the "acl-policy-server-node" policy
Hi everyone, after deploying the consul server and one "client" in my environment, I can see that the client don't reach to join the server...
I got from the client:
I got from the client:
Dec 05 14:46:10 kubetmplp consul[3325]: 2022-12-05T14:46:10.876Z [INFO] agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
Dec 05 14:46:10 kubetmplp consul[3325]: agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
Dec 05 14:46:10 kubetmplp consul[3325]: 2022-12-05T14:46:10.876Z [INFO] agent: started state syncer
Dec 05 14:46:10 kubetmplp consul[3325]: 2022-12-05T14:46:10.876Z [INFO] agent: Consul agent running!
Dec 05 14:46:10 kubetmplp consul[3325]: 2022-12-05T14:46:10.876Z [WARN] agent.router.manager: No servers available
Dec 05 14:46:10 kubetmplp consul[3325]: 2022-12-05T14:46:10.876Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
Dec 05 14:46:10 kubetmplp consul[3325]: agent: started state syncer
Dec 05 14:46:10 kubetmplp consul[3325]: agent: Consul agent running!
Dec 05 14:46:10 kubetmplp consul[3325]: agent.router.manager: No servers available
Dec 05 14:46:10 kubetmplp consul[3325]: agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
consul.hcl
{
"datacenter": "iplan",
"data_dir": "/var/lib/consul",
"encrypt": "3ZYt2575ONn/EYcnQTGKBg==",
"retry_interval": "10s",
"enable_script_checks": false,
"disable_update_check": true,
"dns_config": {
"enable_truncate": true,
"only_passing": true
},
"enable_syslog": true,
"leave_on_terminate": true,
"log_level": "trace",
"rejoin_after_leave": true,
"tls": {
"defaults": {
"verify_incoming": false,
"verify_outgoing": false
}
}
}
Anyone have experience with external services (with health checks) automatically deregistering when registered through Terraform? We've set "deregister_critical_service_after" to an extremely high value, but this external service is still deregistered after some time. The health check is green after initial registration and we use ESM to monitor.
We have consul deployment in k8s using external consul servers. Get
failed to switch to Consul server \"xx.xx.xx.xx:8502\": target sub-connection is not ready (state=TRANSIENT_FAILURE)"}when it tries to connect to the server during upgrade to consul chart 1.0.2 with consul 14.2. Think this issue is due to TLS encryption.
Hello everyone, running into a weird issue with consul-connect-injector with openshift. I am using the helm chart to install it and have the global setting for openshift enabled. This weird thing that that the consul-connect-injector pod never goes healthy. Keep getting readiness probe failed however if I got onto the pod itself and curl the health check it comes back with ok. What am I missing?
Hi is there anyway to set the idle_timeout on TCP services? I see the latest version of consul added
local_idle_timeout_ms but the docs say it is only for HTTP. I see there is this open issue hashicorp/consul#8521. Are there any workarounds or plans to implement it?
I'm having DNS access issues on one of my consul nodes. I've set this acl policy on each node in my cluster, only changing the name accordingly:
agent "blockypi" {
policy = "write"
}
node "blockypi" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
# only needed if using prepared queries
query_prefix "" {
policy = "read"
}The above policy works on all other nodes except blockypi. I have the above policy set on a token which I set as both the default and agent tokens on blockypi, but doing a lookup like dig consul.service.consul @127.0.0.1 -p8600 fails to return any addresses. The same lookup works perfectly fine on my other nodes, with equivalent policies.
The strangest thing about this is that if I temporarily set the default token to a management token then the DNS lookups work. But why DNS doesn't work with the node token breaks my head since all my nodes use the same policy rules as mentioned. I tried removing the policy and token and then recreating it and resetting it on the agent, but the problem remains.
Actually I now also found that another node
nas is having trouble looking up all addresses; dig consul.service.consul @127.0.0.1 -p 8600 +short should return 3 addresses but nas only gets one (its own). Has there been some kind of change in recent Consul versions regarding this? Because this has been working for quite some time until now.
I create my agent tokens with Terraform like this:
resource "consul_acl_token" "agent_token" {
for_each = toset(local.nodes)
description = "Agent token '${each.value}'"
policies = [consul_acl_policy.node_policy[each.value].name]
local = true
}
I'm also not sure why it's trying it's "default.default.B" while the cluster with that service is the A one
however I saw that connecting from A to B does the same, shows
exported~...default.A
Hi Guys,
We have a lot of services running which are accessible through REST API calls. Those calls are authenticated using Basic Auth. We also use HTTPS protocol. These services are docker containers, proxied by Traefik on each node separately and located in customer environment. Consul Server is installed in centrally in our environment in K8S. We are going to use Consul Catalog for dynamic targets discovery in Prometheus.
For example, https://example-service or https://example-service/metrics
We want to include these external services into Consul Catalog via API calls to /agent/service/register or /catalog/register endpoints. But after some days of investigating and tests, it looks like there are no ways to use HTTPS and Basic Auth into the JSON Request Body. Only services with plain HTTP and without basic auth are working as expected. I saw several related questions at stackoverflow, but all of them without answers.
We have a lot of services running which are accessible through REST API calls. Those calls are authenticated using Basic Auth. We also use HTTPS protocol. These services are docker containers, proxied by Traefik on each node separately and located in customer environment. Consul Server is installed in centrally in our environment in K8S. We are going to use Consul Catalog for dynamic targets discovery in Prometheus.
For example, https://example-service or https://example-service/metrics
We want to include these external services into Consul Catalog via API calls to /agent/service/register or /catalog/register endpoints. But after some days of investigating and tests, it looks like there are no ways to use HTTPS and Basic Auth into the JSON Request Body. Only services with plain HTTP and without basic auth are working as expected. I saw several related questions at stackoverflow, but all of them without answers.
- Is it possible to register services to Consul, which is protected via HTTPS and basic auth?
- Are there any other architectural decisions how to securely include standalone protected services to Consul Catalog?
Thanks!