For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.
I don’t understand what’s going on with this one node, it’s the same hardware / config / consul version as all the other working nodes
any suggestions on why this member is flipping in and out of cluster ?
one interesting thing which I had no idea about until now as I’m reading that after 2 unsucessful communications from the broken node, I see a 3rd failed attempt on what appears to be a random port
Jun 21 07:53:55 odo consul[29811]: 2021-06-21T07:53:55.616Z [WARN] agent.client.memberlist.lan: memberlist: Got ping for unexpected node archer.no-dns.co.uk from=127.0.0.1:33718
I get what the error messages are saying mostly, and it makes sense, the broken node is failing the healthchecks so dropping out of the cluster, the working node is responding that it’s getting requests from a non-member of the cluster, but I don’t understand why one node out of 22 is failing the health checks and flip/flopping it’s status
the broken node has nothing wrong with it, resources free on cpu/ram/disk - not under load, no sign of the network flapping in any way etc, from the OS and consul config point of view, everything is fine
the only other point of interest is that occasionally I get the warning Jun 21 08:00:48 archer consul[1720]: 2021-06-21T08:00:48.500Z [WARN] agent.client.memberlist.lan: memberlist: Refuting a suspect message (from: archer.no-dns.co.uk)
from what I’ve read this is because the node left the cluster in an unhealthy state - I’m currently looking how to clean this up to remove confusion
however the single broken node that shows 127.0.0.1:3801
eg: odo.no-dns.co.uk 10.11.216.91:8301 alive client 1.9.6 2 bathstable <default>
paris.no-dns.co.uk 10.11.216.64:8301 alive client 1.9.6 2 bathstable <default>
picard.no-dns.co.uk 10.11.216.151:8301 alive client 1.9.6 2 bathstable <default>
paris.no-dns.co.uk 10.11.216.64:8301 alive client 1.9.6 2 bathstable <default>
picard.no-dns.co.uk 10.11.216.151:8301 alive client 1.9.6 2 bathstable <default>
archer.no-dns.co.uk 127.0.0.1:8301 failed client 1.9.6 2 bathstable <default>
why would the ‘bad’ node be referencing 127.0.0.1
I’m wrondering if I should do a force leave on the node and then see if it rejoins
else "rm -rf" the data dir for consul and reboot the node
do you have any config magic to set machine node ids? or do you let them be auto generated?
Hi Team,
I was trying to create tf state in GCP with version 0.15.4 but terraform validate command is throwing below error.
erraform has been successfully initialized!
$ terraform validate -var-file=environment/${CI_ENVIRONMENT_NAME}/variables.tfvars
╷
│ Error: Failed to parse command-line flags
│
│ flag provided but not defined: -var-file
╵
For more help on using this command, run:
terraform validate -help
Cleaning up file based variables
Please can someone help me, if something i'm missing or is there any change in 0.15.4 version
btw: this is the Consul lobby :smiley:
I'm actually in the middle of a vault rollout, images were built 4 minutes after the binaries were up on docker hub.
I noticed that I sometimes get this in consul logs on consul servers and clients:
"2021-06-30T08:10:26.559Z [WARN] agent: grpc: addrConn.createTransport failed to connect to {192.168.20.43:8300 0 hp-03.als <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp 192.168.20.43:8300: operation was canceled\". Reconnecting..."Any suggestion what this could be? Otherwise, the cluster seems healthy and all nodes are healthy. I use consul 1.10 on all nodes.
My config for a server node looks like this:
"acl": {
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enabled": true,
"token_ttl": "30s",
"tokens": {
"agent": "<redacted>",
"master": "<redacted>",
"replication": "<redacted>"
}
},
"addresses": {
"dns": "127.0.0.1",
"grpc": "127.0.0.1",
"http": "127.0.0.1",
"https": "127.0.0.1"
},
"advertise_addr": "192.168.20.41",
"advertise_addr_wan": "192.168.20.41",
"auto_encrypt": {
"allow_tls": true
},
"bind_addr": "192.168.20.41",
"bootstrap": false,
"bootstrap_expect": 3,
"ca_file": "/etc/consul/certs/consul-agent-ca.pem",
"cert_file": "/etc/consul/certs/server.pem",
"client_addr": "127.0.0.1",
"connect": {
"enabled": true
},
"data_dir": "/data/consul",
"datacenter": "als",
"disable_update_check": false,
"domain": "consul",
"enable_local_script_checks": false,
"enable_script_checks": false,
"encrypt": "<redacted>",
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"key_file": "/etc/consul/certs/server-key.pem",
"log_file": "/var/log/consul/consul.log",
"log_level": "INFO",
"log_rotate_bytes": 0,
"log_rotate_duration": "24h",
"log_rotate_max_files": 14,
"performance": {
"leave_drain_time": "5s",
"raft_multiplier": 1,
"rpc_hold_timeout": "7s"
},
"ports": {
"dns": 53,
"grpc": 8502,
"http": 8500,
"https": -1,
"serf_lan": 8301,
"serf_wan": 8302,
"server": 8300
},
"primary_datacenter": "als",
"raft_protocol": 3,
"recursors": [
"8.8.8.8",
"8.8.4.4"
],
"retry_interval": "30s",
"retry_interval_wan": "30s",
"retry_join": [
"192.168.20.41",
"192.168.20.42",
"192.168.20.43"
],
"retry_max": 0,
"retry_max_wan": 0,
"server": true,
"tls_min_version": "tls12",
"tls_prefer_server_cipher_suites": false,
"translate_wan_addrs": false,
"ui": false,
"verify_incoming": true,
"verify_incoming_https": false,
"verify_incoming_rpc": false,
"verify_outgoing": true,
"verify_server_hostname": true
Has anyone here implemented consul mesh gateway across DCs (or multicloud) in prod: https://www.consul.io/docs/connect/gateways/mesh-gateway
https://www.consul.io/docs/connect/gateways/mesh-gateway/wan-federation-via-mesh-gateways
https://learn.hashicorp.com/tutorials/consul/service-mesh-gateways?in=consul/developer-mesh
5 replies
I'm asking because I have a checkid error with "CheckID XYZ... does not have associated TTL" in the log
but AFAIK, I've got a timeout set on all my checks
hi all :) I'm seeing some very weird behaviour from my consul cluster where http-based health checks are failing with timeouts. It starts happening after a few minutes of a service being registered and won't stop. Seems to be happening with consul 1.9.6 and 1.10.0. Other kinds of checks seem ok. Anyone got any ideas?
@blake @angrycub @anyone_else could you check this and help with suggestions, if any:
https://discuss.hashicorp.com/t/generic-locker-to-give-identity-to-machines-in-an-aws-asg/26289
https://discuss.hashicorp.com/t/generic-locker-to-give-identity-to-machines-in-an-aws-asg/26289
Hi there, I am trying to set up a simple sidecar proxy with envoy. I got node-exporter and an http service (name=web) on separate machines. When I start both sidecars, I am checking with curl if I can reach node-exporter over the proxy, but envoy doesn't follow the upstream in
When I set up the same configuration on the same machine -- it works fine.
What am I doing wrong?
connect.sidecar_service. Instead it forwards all http requests to the actual web service.When I set up the same configuration on the same machine -- it works fine.
What am I doing wrong?
With consul connect, is there a way to know which service made a request to my service? TLS is terminated, so I don't get the certificate but I also don't see any headers being injected which would tell me. Is there any configuration that would allow me to enable e.g. envoy's XFCC header?
1 reply