hashicorp-consul/Lobby

For complex issues please use https://discuss.hashicorp.com/c/consul/, https://github.com/hashicorp/consul/issues or https://groups.google.com/forum/#!forum/consul-tool.

Yann Huissoud

@aiqency

Similar to this issue: hashicorp/consul#1545 I want to add tags to services programmatically in go. /v1/catalog/register is expecting the full service definition (api.AgentServiceRegistration), but I couldn't find any API to get it in the first place (in order to modify it).
/v1/catalog/service/ also doesn't return all required values.

2 replies

I forgot to specify that this must be done remotely. Basically some code on a machine would query the catalog and update the service tags.

iluciv

@iluciv:matrix.org

[m]

Hey there I needed to set up dns with iptables forwarding on the mangaement servers following here https://learn.hashicorp.com/tutorials/consul/dns-forwarding?utm_source=consul.io&utm_medium=docs#iptables-setup

would I need to apply these nat forwarding rules on both the consul agents running in client mode and the consul server cluster agents

at the moment I only have it set on the management server cluster

the clients are running as non root and have the port set to 8600 in their config so I'm assuming that they will need the nat forwarding also. Management nodes are in a forward look up zone in our dns.

iluciv

@iluciv:matrix.org

[m]

oh we have fabio running on the worker nodes (those with consul running in client mode) does that constitute a relay host? (not that flash on dns)

chrisvanmeer

@chrisvanmeer:matrix.org

[m]

Depends on your DNS setup I guess...in my setup, both Consul servers and clients have dnsmasq installed so that they will default query their own DNS instance. The rest of the servers in a separate VLAN use the standard DNS servers of the environment, which has a conditional DNS forwarder for .consul that forwards to the 5 Consul servers.

In any way, it would be beneficial when Consul clients would be able to resolve the .consul domain.

iluciv

@iluciv:matrix.org

[m]

Thanks chrisvanmeer

Ilya Balashov

@ibalashov24

Hello! I have a flood of the following nasty warnings on my Consul installation (v.1.11.3):

[WARN]  agent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=sth-with_underlines

Unfortunately, in my case, service renaming is not feasible. At the same time, my setup does not use the DNS interface at all, so complete DNS disabling would be an appropriate solution, I think. I've tried to set a negative DNS port as suggested here: hashicorp/consul#3135 using CLI flag "-dns-port -1" , but it seems to have no effect.
Could you please advise if there is any way to disable DNS (or solve the warning problem)?

1 reply

Narendra Patel

@narendrapatel

Hi, we had certificates expiry for 2 of our connect enabled services ( 1 is on VMs and other is on K8s). Consul should have renewed them post the 72 hours window but it didn't.
We had to restart the service in VM and recreate 1 pod replica for the K8s service to solve the issue. Have avoided restarting the other replica for time being on K8s to further debug the issue.

4 replies

Narendra Patel

@narendrapatel

One question on monitoring envoy sidecars. Envoy exposes a lot of metrics at the all levels like Listener, Cluster, HTTP level stats, Server level stats, etc.. What should be a good set of key metrics to monitor and perhaps alert on. Istio provides a set of precanned dashboards for use. Do we have some thing similar for Consul?

1 reply

Ryan Goltry

@grogsaxle

Is there a setting to tell consul to log DNS requests? Not that I'm sold on logging DNS requests on consul, there are other options, but wondering if dns logging in consul is an option.

Michael Aldridge

@the-maldridge

this sounds like an XY problem @grogsaxle what is the problem you're trying to solve?

5 replies

Alex Oskotsky

@aoskotsky-amplify

Does terminating gateway support connecting to backends that use TLS? I am trying to put an AWS API Gateway behind a Consul terminating gateway

1 reply

iluciv

@iluciv:matrix.org

[m]

In a consul cluster should you be setting up the worker nodes to be pointing to the consul server cluster as the dns servers? So for example in address block have the 3 consul server nodes? should the consul server node if using iptables be the only ones with recursers on them or should all node have recursers statement in the config?

2 replies

using port 8600 running consul as non root user in this instance

Mauricio Dantas

@MauricioIsARed_twitter

1 reply

did anyone had problems while running consul from systemd?

Michael Aldridge

@the-maldridge

that looks like a pretty standard permissions issue, can you double check that that directory is owned by the expected user?

ShellFu

@shellfu

Hey all, we're running consul on kubernetes. We had to rotate our kubernetes certificates and everything came up fine Consul wise after the restart, however all of the consul-connect-inject sidecars cannot start due to x509 unknown authority "ca".

I restarted the agents and servers again but this did nothing and am about to attempt the cert rotation process documented.

Anyone experience this after rotating k8s certs?

1 reply

Michael Aldridge

@the-maldridge

@blake this page appears to be out of date or otherwise incorrect: https://www.consul.io/docs/connect/configuration it specifies its not required to specify connect { enabled = true } on clients, however without this specified nomad fingerprints the node as attr.consul.connect = false is this a nomad bug or a consul docs bug?

10 replies

Daniel Kimsey

@dekimsey

I have an ingress-gateway with a service-router to split the L7 traffic (following the docs for HTTP listener with Path-based Routing). But the envoy instance only ever reports "no healthy upstreams".
Curiously, envoy /clusters shows all the configured upstream clusters (0 on all stats) and /config_dump shows all the routing config looking sane. I'm not 100% clear on what intentions should be set (ingress name -> router or ingress-name -> final destination), but I've currently got a wildcard destination and it's having no effect. And even then I'd expect a 403 response there.
Logs clearly showing it selecting the configured final-destination cluster (the destination after the service-resolver work is done) and then complaining there are no healthy upstreams. When I look at them in /clusters, I see the correct destination IPs (mesh-gateways) are listed.

I'm at a loss as to why envoy might be considering the clusters to have no healthy upstreams here.

3 replies

Alex Oskotsky

@aoskotsky-amplify

Does failover with service-resolver configs support failing over to the closest datacenter if i provide a list of data centers or will it randomly pick from all of them?

1 reply

Johan Forssell

@johanforssell

This red line - how would I go about listing this from the CLI ?

I.e. - I would like to know which intentions I've missed without going through the web UI

consul intention list does not show missing intentions

iluciv

@iluciv:matrix.org

[m]

I posted this in nomad meant to post it here; Hi there I've inherited some environments, for dns I'm trying to understand why the people before me set the ip in the addresses stanza for dns on the worker nodes (both windows and linux) but on the management nodes they've set is 0.0.0.0. Whats would be the purpose of putting host dns field in there instead of just leaving it as 0.0.0.0 and defaulting to local. Having it this way means for dig queries etc you need to do dig @172.x.x.x -p 8600 some.address ALL instead of just doing dig @localhost

1 reply

Michael Aldridge

@the-maldridge

if I have a repeated block in my configuration that only changes by label, can I have consul-template iterate over a static list?

Daniel Kimsey

@dekimsey

We are using mesh-gateways to route traffic between our DC's and I'm noticing if there are say 3 instances of a service and 2 are marked as unhealthy (failed_eds_health), the mesh-gateway is still forwarding traffic to them resulting in 1/3 success rate for connections. Even though it's well aware they are offline/dead. Does anyone know if that is an intentional design?

Daniel Kimsey

@dekimsey

Relatedly, what would cause a mesh-gateway endpoint to show up as failed_eds_health in a sidecar's clusters output? For the life of me I cannot figure out what is causing this.

Pierre Souchay

@pierresouchay

@the-maldridge if you are doing complicates stuff, you might have a look at https://github.com/criteo/consul-templaterb

Michael Aldridge

@the-maldridge

@pierresouchay I was doing this entirely inside consul-template because I am in a nomad template block for this particular item. I eventually solved it by making this simpler and templating from a different perspective.

[X]

@ennetech

Hi to all, in a two datacenter situation in which now all servers have a dedicated public ip, how wan gossip should be configured?

Alex Oskotsky

@aoskotsky-amplify

Is there a way to override the stream_idle_timeout value in envoy through consul?

Johan Forssell

@johanforssell

Is there a way to check which Agent token my Consul client X is currently using?
I'm setting my agent token with consul acl set-agent-token agent bla-bla-ba.

2 replies

HelloWood

@helloworlde

Hi, I'm new to mDNS, I found the repo https://github.com/hashicorp/mdns, and I want add a dns for a Java server with port 8080 like:

info := []string{"Demo service"}
service, _ := mdns.NewMDNSService("demo", "_http._tcp", "", "", 8080, nil, info)
server, _ := mdns.NewServer(&mdns.Config{Zone: service})
defer server.Shutdown()

After startup mDNS service, I can't request the dns demo._http._tcp.local, how should I add correct dns for this? Thanks very much.

ShellFu

@shellfu

Hey everyone,

I have a mesh gateway federated consul setup utilizing 3 k8s clusters using consul 1.11.3

I have setup a service (service-foo) in each datacenter. In the primary dc I have also setup a service-resolver using a crd to failover to another dc like so:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
  name: service-foo 
spec:
  connectTimeout: 1s
  failover:
    '*':
      datacenters:
        - dc1
        - dc2
        - dc3

Made request to service-foo in dc1, returns 200
Took service-foo down in dc1
Made request to service-foo in dc1, returns 503

I'd have expected Consul to route my request to the next closest with a healthy instance of service-foo.

I have also tried specifying a default subset in the resolver but that did not change the result. The only way I have been able to get to another datacenter is by using redirect in the service-resolver but that is not what is desired here.

Any help would be appreciated.

3 replies

Alvin Lin

@alvinlin123

I know this is not the place for Hashicorp's Memberlist question, but I am wondering where can I get some support on https://github.com/hashicorp/memberlist ? Is there a generic Hashicorp Gitter?

1 reply

Saddam ZEMMALI

@saddam.zemmali_gitlab

HI ALL,
Please I need help to change the datacenter name in a running consul cluster

4 replies

Peter Borghard

@peterborghard

hey folks, anyone know how to add a header in consul connect? I'm able to match and set the destination. But can't seem to figure out the config for header manipulation.

3 replies

Peter Borghard

@peterborghard

Anyone know of any good docs on setting up an http2->grpc proxy using consul connect? I can't seem to get it working, tried a few different ways.

3 replies

Markus Keil

@thereapman

Hi All,
I'm facing failed_eds_health issues between a sidecar and a service in another data center using Meshgate WAN federation.
the sidecar sits in a K8s deployment that has its consul servers externally (no consul srvs inside k8s).
Accessing the service works fine from a non-k8s sidecar on a VM in the same DC as the k8s.
The Helm chart config reference doesn't give any clues on what i'm missing.
Is there a way to debug the health check path or any other leads i could follow?

2 replies

Blake Covarrubias

@blake

Have ideas or need some troubleshooting assistance for Consul? Meet with us and tell us all about it! Sign up at https://hashicorp.sjc1.qualtrics.com/jfe/form/SV_b7cNuNBMrPr4b8q.

madhucs

@madhucs:matrix.org

[m]

Need help , Trying to run CONSUL on EKS (Kubernetes cluster) , I see pods are not comming up the exception I see for pod is below ->

2022-04-27T21:40:58.980112134Z
==> failed to parse /consul/config/..2022\_04\_27\_21\_29\_57.194001731/server.json: 1 error occurred:
\* invalid config key auto\_reload\_config

This is my configuration, deploying through HELM 3+
client:
enabled: false
nodeSelector: |
dev/group: tools
tolerations: |
- key: "tools"
operator: "Equal"
value: "true"
effect: "NoSchedule"
global:
datacenter: adapt
name: consul
server:
enabled: true
extraConfig: |
{
"dns\_config": {
"service\_ttl": {
"\*": "15s"
},
"node\_ttl": "5s",
"max\_stale": "5m"
}
}
image: consul:1.10.3
nodeSelector: |
adapt/group: tools
resources:
limits:
cpu: 300m
memory: 300Mi
requests:
cpu: 300m
memory: 300Mi
tolerations: |
- key: "tools"
operator: "Equal"
value: "true"
effect: "NoSchedule"
ui:
enabled: true
service:
annotations: |
service.beta.kubernetes.io/aws-load-balancer-name: mynlb
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags:Environment=Dev
type: LoadBalancer

3 replies

madhucs

@madhucs:matrix.org

[m]

Woohoo @blake Consul 1.12.0 worked , thank you very much it saved lots of my time !! :-)

Blake Covarrubias

@blake

You're welcome. :-)

Alex Oskotsky

@aoskotsky-amplify

I am sporadically getting these errors upstream connect error or disconnect/reset before headers. reset reason: connection termination: 0 when using consul connect. Is there a workaround for this? I see that Envoy has an option to retry on reset connections. Is that configurable in Consul?

18 replies

Where communities thrive

People

Repo info

Activity