hashicorp-consul/Lobby

For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.

Matt Darcy

@ikonia

2021-08-27T17:26:16.642Z [WARN] agent.server.memberlist.lan: memberlist: Got ping for unexpected node 'archer.no-dns.co.uk' from=127.0.0.1:8301
2021-08-27T17:26:17.144Z [WARN] agent.server.memberlist.lan: memberlist: Got ping for unexpected node archer.no-dns.co.uk from=127.0.0.1:56934
2021-08-27T17:26:17.144Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: EOF

Matt Darcy

@ikonia

is there a way to refute any possible network comms, the node that’s failing is a raspberry pi, with no software firewall, running on a switch port connected to all the other nodes in that list with one or two minor exceptions, so for most of the nodes there is nothing in between the nodes

the only thing I can think of, is that the dead message is from an earlier time / failure of some sort, and because the mesage being refuted it’s stopping it fully joining the cluster, but that doesn’t explain why it’s the only node being referenced on 127.0.0.1

pablo platt

@pablopla_twitter

Is it possible to connect a server on remote datacenter to the service mesh without federation and gateway?

federation will force me to add another Consul cluster and the gateway will be another point of failure

the remote data center has only a single server and adding a Consul and gateway just for one server is too much overhead

6 replies

josuemotte

@josuemotte

hello after a yum update on my environment ( push by automation ) I'm unable to recover my consul cluster , the only error I'm getting is the following :

{"@level":"info","@message":"Request cancelled","@module":"agent.http","@timestamp":"2021-08-30T20:11:12.216779Z","error":"No cluster leader","from":"127.0.0.1:52082","method":"GET","url":"/v1/operator/raft/configuration"}

and

{"@level":"error","@message":"failed to make requestVote RPC","@module":"agent.server.raft","@timestamp":"2021-08-30T20:11:10.433318Z","error":"EOF","target":{"Suffrage":0,"ID":"bas25486-e913-f023-8493-91a46cab6f0a","Address":"10.10.1.4:8300"}}

3 replies

Shai Ben-Naphtali

@shai

What is the variable type of discovery_max_stale? Is it a binary? A string or an int?

The docs don't mention this AFAIK

1 reply

Shai Ben-Naphtali

@shai

Found it in agent/config/config.go

    DiscoveryMaxStale                *string                  `json:"discovery_max_stale" hcl:"discovery_max_stale" mapstructure:"discovery_max_stale"`

Matt Darcy

@ikonia

I’ve got one consul cluster member that’s misbehaving in so many ways, I’m a bit overcome with ‘fault’ on, I’m trying to zero on on one fault/symptom at a time to understand the root problem,

at the moment, if I do a consul members -detailed I see all the cluster members, all correctly reporting their IP address and port 8301 except for the faulty one, which is reporting as 127.0.0.1:8301

the config.jason for that node is not using 127.0.0.1 it’s using it’s correct IP

I cannot understand what is causing the reference to 127.0.0.1 for this one node

Matt Darcy

@ikonia

interesting, I’ve restarted consul 10+ times (while testing stuff) and it’s not made a differnce, the last restart (which was a reboot of the box due to a non-consul package update) a few minutes ago has fixed this and it’s now reporting it’s correct IP address, even though no config file change was made

Alex Oskotsky

@aoskotsky-amplify

Hi. Is it possible to add extra envoy filters while using L7 traffic management features? I see this page https://www.consul.io/docs/connect/proxies/envoy#escape-hatch-overrides says that envoy_listener_json can't be used when using a service-resolver. Is there any workaround to that? I'm trying to use a filter to return a custom response when there are no upstreams available

Frederik Bosch

@frederikbosch

How can I know I successfully enabled consul connect in my cluster?

7 replies

lokesp11

@lokesp11

Hello All,

I need you expert comments on below error:

2021-09-08T09:18:07.191Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: EOF
2021-09-08T09:18:54.191Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: EOF
2021-09-08T09:21:29.190Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: EOF
2021-09-08T09:22:25.191Z [ERROR] agent.server.memberlist.lan: memberlist: Failed fallback ping: EOF

My network connectivity is fine with all the servers in cluster.
I am able to get server members and not facing any other issues but its still keep on logging this error message which I am unable to understand.

Please suggest whats happening here?
Thanks in advance

6 replies

Horizon Zero James

@foozmeat_twitter

Hello, I'm in the process of migrating our web apps to consul/nomad from static configs managed by ansible. We're using HAProxy and I'm planning on sticking with that for now. If I run HAProxy in Nomad how do people manage routing traffic to it given it's IP can change? Do people usually force a static IP on the container? Thanks

Michael Aldridge

@the-maldridge

@foozmeat_twitter you have a few options. The easiest is to create a datacenter in nomad specifically for running haproxy and then run it as a system job. This is a pattern that many of the people over in the nomad room use already. At the more complex end of the scale you could do bgp anycast from any of your nodes running haproxy and then just move the IPs themselves around

8 replies

nivekuil

@.:nivekuil.com

[m]

is there a way to dial a service instance directly through a sidecar proxy, without transparent proxy? can't use the iptables traffic redirection as I have multiple services in the same netns

4 replies

3nprob

@3nprob

Does anyone have any hints for requirements/troubleshooting WRT TLS for WAN gossip?
I have one existing Consul cluster (intended as new primary dc) and adding a new one.
Primary one set up with built-in CA. New one set up using a separate CA (Vault, FWIW)
I have attempted a join -wan and the primary cluster sees the new cluster server as a gossip node
But both ends are giving TLS errors.
Servers for both DCs have the CA for the server certificates added as trusted CA certs on the system (on debian, using update-ca-certificates)

I don't seem to be able to find any settings in Consul to add additional CAs, or separate CAs for WAN.
Is there any way to get this working or is there a requirement that server TLS certs for all servers in a WAN gossip pool come from the same CA?
(consul members on server in new DC is yielding 403 ACL not found but I guess that is expected until replication is successful and that the cert issues are preventing replication)

3nprob

@3nprob

This verifies fine on both ends: openssl s_client -showcerts -verify 5 -connect server.$OTHER_DC.consul:8300 < /dev/null (using /etc/hosts to map hostname to server in the other DC)

3nprob

@3nprob

Stll, I get on server in old, primary DC

[ERROR] agent.server.rpc: failed to read byte: conn=from=$PROXY_IP :32139 error="remote error: tls: bad certificate"

and on new :

[WARN]  agent.server.replication.acl.role: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL roles: rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"
[ERROR] agent.server.connect: error performing intention migration in secondary datacenter, will retry: routine="intention config entry migration" error="rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"
[ERROR] agent.server.rpc: RPC failed to server in DC: server=$IP:8300 datacenter=$PRIMARY_DC method=ConfigEntry.ListAll error="rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"

3nprob

@3nprob

Probably unrelated but running consul acl bootstrap on new server return Failed ACL bootstrapping: Unexpected response code: 500 (ACL support disabled) (ACL enabled in config , only hardcoded token is replication as per docs)

3nprob

@3nprob

OK, that turned out to be way easier than expected. Had missed the config option ca_path, changing to that and hoisting CAs in there works fine. No need to involve the system trust chain

Miguel Araujo

@Maikuh

Hello. I'm trying to use Consul with Kubernetes (minikube). I'm trying to use the CRDs for Service Intentions, yet when I apply them, I get the following error

failed calling webhook "mutate-serviceintentions.consul.hashicorp.com": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block

I followed this tutorial and I get the error both with L4 and L7. Note that via the UI and the API it works, just the CRDs don't.

Miguel Araujo

@Maikuh

It seems that the above happens mostly if all consul services are not ready yet (probably because the CRDs are actually just making HTTP calls to Consul's API, which would make sense). This is an issue since I'm using tools such as Tilt and they create the CRDs at the same time it installs the helm charts. Basically, I'd only be able to make it work if I install the CRDs manually through the terminal.

Is this a limitation of Consul's CRDs implementation? I've used CRDs from other solutions before (Gloo IIRC) and have been able to install them at the same time as other helm charts and resources, using Tilt, with no issues.

pablopla

@pablopla

Do I have to use ingress gateway to interact with clients outside of the consul cluster?

or can I just expose a public IP?

1 reply

Michael Aldridge

@the-maldridge

can anyone on a relatively recent ubuntu confirm that https://learn.hashicorp.com/tutorials/consul/dns-forwarding#systemd-resolved-setup works?

I can only reliably get it to give me the following error:

Sep 15 20:22:46 ip-10-2-26-216 systemd-resolved[19974]: /etc/systemd/resolved.conf.d/consul.conf:1: Assignment outside of section. Ignoring.
Sep 15 20:22:46 ip-10-2-26-216 systemd-resolved[19974]: /etc/systemd/resolved.conf.d/consul.conf:2: Assignment outside of section. Ignoring.

19 replies

Mike Cardwell

@mike.cardwell:grepular.com

[m]

Does a consul client need to be in the same datacenter as a consul server?

2 replies

lokesp11

@lokesp11

We have Consul Cluster on VM's and we have agent deployed on VM's and K8's both:
Its been working fine but recently we saw an issue.

Due to OS upgrade one of server was down and all the vm's got in sync with current server peers but somehow agent deployed on k8s was still trying to connect to same server which was down for patching. There was some delay we saw in getting latest state of servers on k8s.

2021-09-10T08:03:59.183Z [ERROR] agent.client: RPC failed to server: method=KVS.List server=**.**.**.**:8300 error="rpc error making call: rpc error getting client: failed to get conn: dial tcp <nil>->**.**.**.**:8300: connect: connection refused"

Is there any setting in helm chart which can help in immediate sync and avoid this issue?

sstent

@sstent

@lokesp11 what is your retry-join pointed at? either a list of nodes to try, load balanced IP, or consul dns entry would be good

lokesp11

@lokesp11

@sstent It's list of nodes . Is there any limitation with list of nodes?
Even if one server is down all server peers/agents will have the info about it leaving the mesh and catalog will also be updated accordingly . I am trying to understand once this info is updated is there any possibility if consul agent will still try to connect the server which not reachable?

johnny101

@johnny101:matrix.org

[m]

We seem to be hitting an issue with token rotation and the consul-agent token in our Nomad cluster. If indeed an issue that needs fixing (rather than some kind of approach change), I'm not sure if this would fall more under Nomad or Consul. Hence the cross post here in addition to the Nomad room. I described the setup and the issue here: https://github.com/hashicorp/nomad/issues/9813#issuecomment-930456285. If anyone has any feedback, that would be appreciated.

Ross Has No Clever Friends

@rosshasnocleve1_twitter

Hashicorp devs, do any of you know someone who works on go-discover? There is a pull request lingering since February that looks approved but not merged, and a lot of people are waiting for it.

Blake Covarrubias

@blake

The Consul team has been reviewing go-discover PRs pretty regularly. Which PR are you referring to?

Michael Aldridge

@the-maldridge

reviewing yes, but merging relatively infrequently. I'd bet this is in reference to either proxmox or hetzner, as those are the only two from February

Ross Has No Clever Friends

@rosshasnocleve1_twitter

HI Blake, this one: hashicorp/go-discover#167

Blake Covarrubias

@blake

I was mobile earlier and was having some trouble replying from my phone. I tried to send a few messages, but it looks like they didn't go through.

Michael Aldridge

@the-maldridge

the gitter/matrix ecosystem has been a little wobbly today, probably a little bit of both

Blake Covarrubias

@blake

I'll discuss this PR with our team tomorrow, and will ask that someone follow up on the GitHub issue.

lokesp11

@lokesp11

Does consul support CA Signed certificate for tls communication and can it be integrated with vault to get certificate from vault pki? We are exploring option to use VAULT PKI Infrastructure and trying to implement consul tls communication with certificate generated by vault pki instead of inbuild consul CA?

Please suggest or help in pointing me to similar use case if exist?

Thanks in advance

Ross Has No Clever Friends

@rosshasnocleve1_twitter

Thanks Blake!

Yann Huissoud

@aiqency

Any updates on hashicorp/consul#8687. Can we access the specific key modified through consul watch keyprefix as of now?

Narendra Patel

@narendrapatel

Hi, has anyone tried the Escape-Hatch Overrides option? Need to configure envoy access logging. Able to configure it directly on envoy but struggling to set it via Consul. Also how are envoy cluster names formed? I see consul configuring something like api.default.dc1.internal.af617b02-1e21-52c2-d297-36b92be86af9.consul. Not sure what does this hexadecimal string signifies.

Where communities thrive

People

Repo info

Activity