Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    John Spencer
    Noob question about service to service communication when services are using random, dynamically allocated ports (like in a Consul/Nomad cluster). If the port is know ahead of time, e.g. port 80, serviceA can talk to serviceB using the dns record serviceB.service.consul:80. But if the port is dynamically allocated, how does serviceA communicate with serviceB? It could find the port via dig SRV or consul API, but that's additional application code and overhead. Is there a better way?
    Yoan Blanc
    @johnnyplaydrums you need some kind of load balancer, e.g. https://www.consul.io/docs/connect
    2 replies
    Sam Lee
    hi, i'd like to access consul ui by ingress subpath.
    deployed "hashicorp/consul:1.10.3" helm chart and tried this additional extraConfig. But it's redirected to "/" path . Is there anything i missed ?
      extraConfig: |
          "ui_config": {
            "content_path": "/sandbox/consul"
    1 reply
    Ross Rochford

    @johnnyplaydrums I suspect a lot of people end up using consul upstreams (sidecars) in Nomad simply for the convenience of it. In this scenario Nomad gives your tasks a single addr/port to communicate through.

    It would be nice to have a similarly convenient setup but with the option of bypassing the security features (encryption, intentions) of consul connect. i.e. just the load balancing and service discovery pieces.

    1 reply
    Andreas Anderssson
    I am having a problem where consul isn't being updated fast enough with pods in kubernetes.
    Not sure how to solve that problem.
    We have consul-write-interval set to 1s.
    Yet, our loadbalancer (traefik) is still trying to send traffic to old pods.
    I am not sure if it's because
    1) We don't execute healthchecks from consul to kubernetes pods.
    2) The data in consul is stale.
    How have other people solved this problem?
    Narendra Patel
    Hi, is it possible to set mTLS for ingress-gateways? We want to set inbound mTLS to the mesh for one of our services. Not able to find options for the same. Saw an option for mTLS with https://www.hashicorp.com/blog/proxy-ingress-to-consul-service-mesh but it involves too many moving parts. Preference is for Ingress Gateway with envoy proxy.
    4 replies
    Alex Oskotsky
    I'm seeing a weird issue where every 10 minutes envoy logs a line this this add 3 cluster(s), remove 2 cluster(s). During that time I'll also see /failed_eds_health for a few seconds when viewing the cluster from the envoy admin UI. The nodes show as healthy in the consul UI during time this is happening. This started happening after upgrading to Consul 1.10. Has anyone ever seen this or have any suggestions?
    2 replies
    Chris Johnson
    @blake in your demo of using transparent proxies, you were JUST about to start typing a curl command so I could finally understand how they decide the service to route to, but then you got sidetracked chasing down problems in the demo hahaha
    Demo of using transparent proxies on VMs, not k8s, that is
    Anyway, it's still unclear to me. What does an application have to be configured to use as the DNS name when making the request? Should the application use http or https? And does the port need to be something specific?
    2 replies
    Right now I'm in the middle of reconfiguring my tooling to produce a local IP and whatever the upstream port is for that service. Using transparent proxies, would it be using the consul DNS name? And would that be the <svc>.service.consul form or <svc>.ingress.consul form?
    J. Gavin Ray
    Hello all!
    Hi all, qq? failed to setup alloc: pre-run hook "network" failed: failed to create network for alloc: Failed to pull gcr.io/google_containers/pause-amd64:3.1: API error (500): Get https://gcr.io/v2/: net/http: TLS handshake timeout we are seeing this when we start a nomad job even though we have pause-amd64 image loaded locally but since one of our env's has strictly no internet access to outside world is there a way in nomad that you can force it to not look into google container registry?
    1 reply

    Hi . I installed consul on K8s with the following command

    helm -n consul-server install --create-namespace -g hashicorp/consul -f consul-values.yaml

    cat consul-values.yaml                                                                                                                   ✔  10:52:28 
      enabled: true
      name: consul
        manageSystemACLs: true
        enabled: true
        enableAgentMetrics: true
      image: "hashicorp/consul:1.10.3"
      imageK8S: "hashicorp/consul-k8s-control-plane:0.36.0"
      enabled: true
      replicas: 1
      enabled: true
      enabled: false
        defaultEnabled: false
      enabled: true
        type: LoadBalancer
      enabled: true

    I opened the Ingress endpoint. However where do i find the token to login to save data under the KV? i always get 403 since i am not logged in

    1 reply
    Yann Huissoud
    From a golang perspective what's the benefit of using hcl:"foo" instead of json:"foo"?
    Yann Huissoud
    basically we are writing our own library and was wondering what is the benefit of using hcl?
    1 reply
    Narendra Patel
    What is the standard way to override envoy configs via envoy_public_listener_json and envoy_listener_json in kubernetes. On VM able to override via sidecar_service block.
    Also how to access envoy config dump from the envoy sidecar in kubernetes
    Ravindra Verma
    Hi experts, I need some help in consul-nomad cluster setup, I have installed the consul cluster on azure Vm's , but its not coming up as a cluster getting error :
    Nov 15 06:37:26 nomad-consul-vm1 consul[29364]: 2021-11-15T06:37:26.941Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
    Nov 15 06:36:18 nomad-consul-vm2 consul[382]: 2021-11-15T06:36:18.676Z [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=
    Yoan Blanc
    check the encrypt config.
    Ravindra Verma
    could you provide a reference , so i can match my file
    Ravindra Verma
    I have installed the consul cluster on azure Vms 3nodes, can anyone suggest how to set up DNS for this? shall I use internal LB or any other method which I can use? Please guide
    Shantanu Gadgil
    Hi @Ravindraverma1 what have you tried so far? Can you post your config files with secrets redacted? What you are asking are too much of open ended questions. I assume by DNS you mean services looking up each other by name? Or some thing else?

    Hello all! Maybe someone can point me in the right direction.
    I am in the process of finalizing a Proof of Concept using Nomad and Consul.
    My remaining issue is with Consul Federation.

    I currently have 2 separate Nomad Clusters, and 2 separate Consul clusters
    I have federates the Consul clusters, and when I use "consul members -wan" I can evidently see that all required Consul server nodes are listed across data centres.

    I have deployed a nomad job (docker http-echo) named "webserver". I have deployed 1 instance of this on Nomad dc1 and Nomad dc2, and registered it to Consul using the following stanza:
    service {
    name = "webserver"
    tags = ["webserver"]
    port = "http"
    meta {
    meta = "Consul Connect Test"

    I used the same service stanza when deploying my job to both Nomad clusters, however, when I login to the Consul UI, dc1 Consul is showing 1 instance of webserver, whilst dc2 Consul is showing another instance of webserver.

    Is there anyway that Consul would be aware that these are in fact yet another replica to the same deployment?

    The idea is to use 1 single source of truth from Consul to integrate with a Load Balancer with AS3.

    5 replies
    I can also select between the different Consul DCs in the web UI - further indicating that federation should be OK
    hello all, I have installed consul in Kubernetes Token page says "you are not authorized 403" how can I fix that problem?
    1 reply
    André Ilhicas

    Hi folks, I'm trying to implement a "hackfix" solution to use consul transparent proxy within nomad, and I'm able to correctly register the connect evoy sidecar proxy and have healthchecks ok etc
    However, I'm trying to curl another connect enabled service and I'm always greeted by the Empty Reply from server, despite having the clusters registered within envoy /clusters endpoint meaning that the outbound traffic grabbed by the proxy A is not correctly using mtls to communicate with the service B

    Anyone tried something similar or faced a similar issue?

    (moving same message I pasted in nomad group)

    1 reply
    Is it expected behavior for a consul agent to return a 500 Internal Server Error response code when trying to deregister a service that is not registered in the agent, through an HTTP API call to /v1/agent/service/deregister/:service-id?
    I'd expect maybe a 400 Bad Request ("can't deregister something that's not registered") or even a 200 Ok ("already deregistered, nothing to do "(idempotency-like)), but 500 is a bit strange.
    4 replies

    export GOPATH=/opt/gows

    git clone'd consul; make tools

    make dev throws the following error

    $ make dev
    ==> Building Consul - OSes: linux, Architectures: amd64
    Building sequentially with go install
    ---> linux/amd64
    cp: cannot stat '/opt/gows/bin/consul': No such file or directory
    ERROR: Failed to build Consul for linux/amd64
    make: * [GNUmakefile:150: dev-build] Error 1

    anyone knows why is this failing ?

    1 reply
    Ravindra Verma
    Hi experts, i have created infra on azure and installed nomad and consul cluster there . could you please confirm to setup consul DNS on clients, i followed few steps but its not working out . can anyone suggest a good way to setup this
    1 reply
    Owen Byrne

    hey folks - i'm beginning to use consul-connect on kubernetes. I want to set forward_client_cert_details: ALWAYS_FORWARD_ONLY as a default in the public and outbound envoy listeners . I'm struggling to find a straightforward way to do this. Would the only option be to use escape hatches: https://www.consul.io/docs/connect/proxies/envoy#envoy_public_listener_json and https://www.consul.io/docs/connect/proxies/envoy#envoy_listener_json ?

    If i were to use the escape hatch approach, would I need to wire it up with things such as the IP address and port number which would otherwise be dynamically configured e.g.:

    dynamic_listeners": [
              "name": "public_listener:",
              "active_state": {
                "version_info": "509d3db3174c07668c164b6772525adbb945e5fcbacaeddacf9364512e06d91b",
                "listener": {
                  "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
                  "name": "public_listener:",
                  "address": {
                    "socket_address": {
                      "address": "",
                      "port_value": 20000

    Any alternative methods to solve this? Could it be configured via bootstrap somehow?

    2 replies
    Hello, I'm looking at using maglev load balancing algorithm to a connect service for consistent load balance hashing. Ideally, I'd like this connect service to act as a active/hot-standby cluster, so that n instances are available, but only 1 is routed to at any given time as long as it stays healthy. I can do this with with maglev load balancing using an HTTP header that is constant -- then all requests are routed to that instance. For this, the requesting app must ensure the appropriate static HTTP header is added.
    I see that in consul 1.11.0-beta3 HTTPHeaderModifiers for request and response headers has been added, which is a great feature. Ideally, I'd like to configure the static header that maglev would consume for consistent hashing in a service-resolver config kind by setting up a "RequestHeader" "Set" to inject that static header in a service-router config kind.

    That way, service consuming apps don't have to worry about setting a header, and if already improperly set, the header would be properly overwritten to maintain the desired active/hot-standby service routing. I can use header injection to see maglev properly working by adding a response header with a value of envoy's %UPSTREAM_REMOTE_ADDRESS% when I manually add a static maglev hashing header, for example, with curl.

    However, if I add that same static maglev hashing header in a service-router config kind which should occur earlier in the consul traffic management than a service-resolver evaluation (routing -> splitting -> resolution), maglev consistent hashing doesn't work. I've checked by tcpdump in the upstream service environment and do see that the maglev static hashing header was properly injected. It just seems like maglev doesn't recognize it when it's injected by the routing HTTPHeaderModifiers instead of manual header addition from the calling app.

    The service I'm using does have subsets, so maybe if I simplify a bit or try a few other things I can get it to work... But before I dump more time into playing around with it, wanted to check if HTTPHeaderModifier injection for maglev hashing is even possible. Anyone have any ideas? Maybe @blake ? Thanks!
    Mark Martirosian
    Hello, getting "memberlist: Failed to resolve i-032e19c9eecd21e.region/[IPv6 address]: lookup [IPv6 address]: no such host" messages. The cluster is otherwise operational. Any idea how to fix?
    Owen Byrne

    Hello, when I apply a ProxyDefaults resource on Kubernetes (1.18), this error appears in the consul-controller logs:

    2021-12-01T13:15:21.855Z    INFO    webhooks.proxydefaults    validate create    {"name": "global"}
    2021-12-01T13:15:21.971Z    INFO    controller.proxydefaults    config entry not found in consul    {"request": "default/global"}
    2021-12-01T13:15:21.978Z    INFO    controller.proxydefaults    config entry created    {"request": "default/global", "request-time": "3.458308ms"}
    2021-12-01T13:15:22.004Z    ERROR    controller.proxydefaults    Reconciler error    {"reconciler group": "consul.hashicorp.com", "reconciler kind": "ProxyDefaults", "name": "global", "namespace": "default", "error": "Operation cannot be fulfilled on proxydefaults.consul.hashicorp.com \"global\": the object has been modified; please apply your changes to the latest version and try again"}

    Here's what I'm applying:

    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
      name: global
        envoy_public_listener_json: "{\"@type\":\"type.googleapis.com/envoy.config.listener.v3.Listener\",\"name\":\"public_listener:\",\"address\":{\"socket_address\":{\"address\":\"\",\"port_value\":20000}},\"filterChains\":[{\"filters\":[{\"name\":\"envoy.filters.network.http_connection_manager\",\"typed_config\":{\"@type\":\"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\"stat_prefix\":\"public_listener\",\"forward_client_cert_details\":\"APPEND_FORWARD\",\"set_current_client_cert_details\":{\"subject\":true,\"dns\":true,\"uri\":true},\"route_config\":{\"name\":\"public_listener\",\"virtual_hosts\":[{\"name\":\"public_listener\",\"domains\":[\"*\"],\"routes\":[{\"match\":{\"prefix\":\"/\"},\"route\":{\"cluster\":\"local_app\"}}]}]},\"http_filters\":[{\"name\":\"envoy.filters.http.router\"}]}}]}]}"

    Error aside, it does appear to have created the resource ok:

    kubectl get proxydefaults.consul.hashicorp.com
    global   True     17m           17m

    This anything to worry about?

    André Ilhicas
    Folks, I just wanted to come here to say thank you so much for the version 1.11 virtual ip feature
    I was able to use it to have transparent proxy in nomad using default dialed directly as false using virtual ips and it just works great.
    So thanks for all the engineering effort in making this possible
    3 replies
    Afternoon, does anybody have any tips on how to debug ingress gateway hosts issues. I am getting a 404 when I use the following
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: IngressGateway
      name: ingress-gateway
        enabled: true
        - port: 8080
          protocol: http
            - name: frontend
              - "frontend.xxx.xxxx.com"
              - "localhost"
    2 replies
    Morning, I am a little confused between ingress gateway and ingress controller for k8s. Seems all the demos use traefik or kong as the ingress controller, but then what is the use of the ingress gateway? Do I need to use both? The reason for asking is that it does not seem possible to have a cert-manager (connected to HashiCorp Vault) to issue a Certificate for the ingress gateway, and that you need to have an ingress controller for that. But If I use something like traefik whats the point of also configuring a gateway with Splitting and Resolver?
    2 replies