For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.
Jul 21 10:10:53 wesley consul[15814]: 2021-07-21T10:10:53.570Z [ERROR] agent.server.rpc: unrecognized RPC byte: byte=71 conn=from=10.11.216.64:52784
it’s prometheus running on the same node, querying the consul servers on the wrong port, consul is expecting RPC and prometheus is scraping with a standard TCP request, hence why it’s unrecognised RPC
Hi all, we've been testing out
consul connect ca set-config to rotate between Vault CA endpoints gracefully. The issue is that while existing proxies work fine during the rotation process, new proxies can't seem to reference the new CA bundle until the Consul leader is restarted and an election is forced. Restarting the leader immediately after setting the config causes old proxies to break for a few minutes, however, so this isn't an option. Has anyone dealt with this before? We are on Consul 1.9.6, Envoy 1.16.4.
Hi all, after upgrading a linux server from U16 to U18 (without touching consul) I'm having the error: Node name XXX is reserved by node YYY. I have tried a number of things (leave, force-leave, use the API to deregister and then register with the new ID..) without result. There is anyway to set the new ID in the cluster? Anything else? We are using consul 1.8.4 Thanks !
(did you not want to move to Ubuntu 20.04 ?
We cannot upgrade to Ubuntu 20 :(
Stopping consul and editing node-id with the old value will solve the problem?
Thanks a lot @ikonia although I´m not sure if this was the correct way to fix the issue
but that file seems to cause lots of problem if the instance changes in some way / replaced with the same hostname
hi all, so we are running consul in heavy scale (around 12K nodes in a cluster). We see something very wired. We have nodes that exists in the nodes catalog but not in the consul members (serf). This is reflected also in the consul UI as those nodes appear not to have serf health check but they do appear in the ui. of course consul agent is running on those hosts and from the agent perspective all is fine, although it is not listed in the members of the cluster. Any help will be great, we are breaking our heads against the wall with this fo long time
Thanks for your replays.
@pierresouchay , our cluster is not that loaded from CPU or Memory perspective.Also calling the catalog deregister is problematic as those nodes are actually running and they think they are part of the cluster. The only valid solution we have found was restarting the node itself, but at our scale checking this is it not trivial. It feels like something is broken when syncing the serf members and catalog if we end up in such cases. @gc-ss Attaching CPU load on the server hosts
@pierresouchay , our cluster is not that loaded from CPU or Memory perspective.Also calling the catalog deregister is problematic as those nodes are actually running and they think they are part of the cluster. The only valid solution we have found was restarting the node itself, but at our scale checking this is it not trivial. It feels like something is broken when syncing the serf members and catalog if we end up in such cases. @gc-ss Attaching CPU load on the server hosts
Deregister the node would not be an issue, the node would register again when anti-entropy would trigger (i would say around 10/15 minutes on such a big cluster). If it doez not, it means something is broken on the node itself, so restarting the node looks like the only valid option. Nothing weird on those nodes logs?
Hey folks - does anyone know how the hexidecimal value at the beginning of the service name that's returned from a dig SRV is generated? Is there anyway to know this value from a service running in consul (via env var, or some deterministic method). For example when you dig for an SRV record, you're given a dns name like ac1f8802.addr.dc1.consul. Where does that ac1f8802 come from, and is it possible to know that from within a service running in consul?
172.31.136.2
Like if I'm a service and want to know that hostname, it looks like I can derive it from <hex_value>.addr.<dc>.<tld>. Is there an easy way to go from IP address -> hex value?
Thank you sir
🙏
I didn't find anything in the docs, so I'm asking here. Does the transparent proxy support Consul's own DNS as well, instead of Kubernetes DNS? We're running multiple clusters, so Kubernetes DNS won't do any good for us. But we do have routeable pod IPs, which means two pods from different clusters can talk to one another.
Is there an automated way to upgrade consul_intentions < 1.9 to the new
Example
consul_config_entry syntax with terraform? I have hundreds of consul intentions and changing all these by hand is toing to take forever.Example
# This was correct in version 2.10.0
resource "consul_intention" "database" {
source_name = "api"
destination_name = "db"
action = "allow"
}
# This is now the correct configuration starting version 2.11.0
resource "consul_config_entry" "database" {
name = "db"
kind = "service-intentions"
config_json = jsonencode({
Sources = [{
Action = "allow"
Name = "api"
Precedence = 9
Type = "consul"
}]
})
}
10 replies
When running consul connect in Nomad with an envoy sidecar, consul agent and envoy sidecar container stderr logs show the following grpc permission related errors below. Anyone familiar with this or how to debug it?
# From consul agent on the host (log level is trace):
agent.envoy.xds: Incremental xDS v3: xdsVersion=v3 direction=request protobuf="{ "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster"
agent.envoy.xds: subscribing to type: xdsVersion=v3 typeUrl=type.googleapis.com/envoy.config.cluster.v3.Cluster
agent.envoy.xds: watching proxy, pending initial proxycfg snapshot for xDS: service_id=_nomad-task-6227f408-bee9-77fa-529f-924164f42b80-group-api-count-api-9001-sidecar-proxy xdsVersion=v3
agent.envoy.xds: Got initial config snapshot: service_id=_nomad-task-6227f408-bee9-77fa-529f-924164f42b80-group-api-count-api-9001-sidecar-proxy xdsVersion=v3
agent.envoy: Error handling ADS delta stream: xdsVersion=v3 error="rpc error: code = PermissionDenied desc = permission denied"
# From envoy stderr in the envoy sidecar container (log level is trace):
DeltaAggregatedResources gRPC config stream closed: 7, permission denied
gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
having an issue " agent.server.memberlist.wan: memberlist: Failed to resolve consul-consul-server-1.dc1/2605::::::8302: lookup 2605:::::::8302: no such host" .. the issue is tthere is no brackets on the ipv6 like [2605:::]8302 . how do i introduce this in the wan lookup .. environment is openshift ipv6 cluster
1 reply
hello. I have a working mesh gateway with wan federation. from both datacenters I can curl /v1/catalog/services?dc=<other-dc> and see the services running there and "consul members -wan" shows servers in both dcs
however, services themselves (e.g. the socat example) cannot connect between the DCs
The only errors I see in the consul logs are on the secondary DC where there are lots of warnings:
Err :connection error: desc = "transport: Error while dialing dial tcp <internal ip of server in primary dc>:8300: i/o timeout"
I outlined the issue here https://discuss.hashicorp.com/t/unable-to-connect-services-between-datacenters-despite-working-mesh-gateways/28721
I would really appreciate any help as I'm completely stuck
1 reply
my home test lab (all running consul 1.10.1) is having an odd problem with one node - it seems to never truly join the cluster properly, I’ve just done a force-remove and then a join wich didn’t error, however the node is filled with errors / problems, I cannot understand the reasoning for the behaviour of this node. The status of the test cluster is as follows.