Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Ravindra Verma
    @Ravindraverma1
    I have installed the consul cluster on azure Vms 3nodes, can anyone suggest how to set up DNS for this? shall I use internal LB or any other method which I can use? Please guide
    Shantanu Gadgil
    @shantanugadgil
    Hi @Ravindraverma1 what have you tried so far? Can you post your config files with secrets redacted? What you are asking are too much of open ended questions. I assume by DNS you mean services looking up each other by name? Or some thing else?
    GOEDP
    @GOEDP

    Hello all! Maybe someone can point me in the right direction.
    I am in the process of finalizing a Proof of Concept using Nomad and Consul.
    My remaining issue is with Consul Federation.

    I currently have 2 separate Nomad Clusters, and 2 separate Consul clusters
    I have federates the Consul clusters, and when I use "consul members -wan" I can evidently see that all required Consul server nodes are listed across data centres.

    I have deployed a nomad job (docker http-echo) named "webserver". I have deployed 1 instance of this on Nomad dc1 and Nomad dc2, and registered it to Consul using the following stanza:
    service {
    name = "webserver"
    tags = ["webserver"]
    port = "http"
    meta {
    meta = "Consul Connect Test"
    }
    }

    I used the same service stanza when deploying my job to both Nomad clusters, however, when I login to the Consul UI, dc1 Consul is showing 1 instance of webserver, whilst dc2 Consul is showing another instance of webserver.

    Is there anyway that Consul would be aware that these are in fact yet another replica to the same deployment?

    The idea is to use 1 single source of truth from Consul to integrate with a Load Balancer with AS3.

    5 replies
    I can also select between the different Consul DCs in the web UI - further indicating that federation should be OK
    tirelibirefe
    @tirelibirefe
    hello all, I have installed consul in Kubernetes Token page says "you are not authorized 403" how can I fix that problem?
    image.png
    1 reply
    André Ilhicas
    @Ilhicas

    Hi folks, I'm trying to implement a "hackfix" solution to use consul transparent proxy within nomad, and I'm able to correctly register the connect evoy sidecar proxy and have healthchecks ok etc
    However, I'm trying to curl another connect enabled service and I'm always greeted by the Empty Reply from server, despite having the clusters registered within envoy /clusters endpoint meaning that the outbound traffic grabbed by the proxy A is not correctly using mtls to communicate with the service B

    Anyone tried something similar or faced a similar issue?

    (moving same message I pasted in nomad group)

    1 reply
    parmsib
    @parmsib:matrix.org
    [m]
    Hello!
    Is it expected behavior for a consul agent to return a 500 Internal Server Error response code when trying to deregister a service that is not registered in the agent, through an HTTP API call to /v1/agent/service/deregister/:service-id?
    I'd expect maybe a 400 Bad Request ("can't deregister something that's not registered") or even a 200 Ok ("already deregistered, nothing to do "(idempotency-like)), but 500 is a bit strange.
    4 replies
    inouthack
    @inouthack
    Hi

    export GOPATH=/opt/gows

    git clone'd consul; make tools

    make dev throws the following error

    $ make dev
    ==> Building Consul - OSes: linux, Architectures: amd64
    Building sequentially with go install
    ---> linux/amd64
    cp: cannot stat '/opt/gows/bin/consul': No such file or directory
    ERROR: Failed to build Consul for linux/amd64
    make: * [GNUmakefile:150: dev-build] Error 1

    anyone knows why is this failing ?

    1 reply
    Ravindra Verma
    @Ravindraverma1
    Hi experts, i have created infra on azure and installed nomad and consul cluster there . could you please confirm to setup consul DNS on clients, i followed few steps but its not working out . can anyone suggest a good way to setup this
    1 reply
    Owen Byrne
    @byrneo

    hey folks - i'm beginning to use consul-connect on kubernetes. I want to set forward_client_cert_details: ALWAYS_FORWARD_ONLY as a default in the public and outbound envoy listeners . I'm struggling to find a straightforward way to do this. Would the only option be to use escape hatches: https://www.consul.io/docs/connect/proxies/envoy#envoy_public_listener_json and https://www.consul.io/docs/connect/proxies/envoy#envoy_listener_json ?

    If i were to use the escape hatch approach, would I need to wire it up with things such as the IP address and port number which would otherwise be dynamically configured e.g.:

    dynamic_listeners": [
            {
              "name": "public_listener:100.96.140.127:20000",
              "active_state": {
                "version_info": "509d3db3174c07668c164b6772525adbb945e5fcbacaeddacf9364512e06d91b",
                "listener": {
                  "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
                  "name": "public_listener:100.96.140.127:20000",
                  "address": {
                    "socket_address": {
                      "address": "100.96.140.127",
                      "port_value": 20000
                    }
                  },

    Any alternative methods to solve this? Could it be configured via bootstrap somehow?

    2 replies
    johnny101
    @johnny101:matrix.org
    [m]
    Hello, I'm looking at using maglev load balancing algorithm to a connect service for consistent load balance hashing. Ideally, I'd like this connect service to act as a active/hot-standby cluster, so that n instances are available, but only 1 is routed to at any given time as long as it stays healthy. I can do this with with maglev load balancing using an HTTP header that is constant -- then all requests are routed to that instance. For this, the requesting app must ensure the appropriate static HTTP header is added.
    I see that in consul 1.11.0-beta3 HTTPHeaderModifiers for request and response headers has been added, which is a great feature. Ideally, I'd like to configure the static header that maglev would consume for consistent hashing in a service-resolver config kind by setting up a "RequestHeader" "Set" to inject that static header in a service-router config kind.
    johnny101
    @johnny101:matrix.org
    [m]

    That way, service consuming apps don't have to worry about setting a header, and if already improperly set, the header would be properly overwritten to maintain the desired active/hot-standby service routing. I can use header injection to see maglev properly working by adding a response header with a value of envoy's %UPSTREAM_REMOTE_ADDRESS% when I manually add a static maglev hashing header, for example, with curl.

    However, if I add that same static maglev hashing header in a service-router config kind which should occur earlier in the consul traffic management than a service-resolver evaluation (routing -> splitting -> resolution), maglev consistent hashing doesn't work. I've checked by tcpdump in the upstream service environment and do see that the maglev static hashing header was properly injected. It just seems like maglev doesn't recognize it when it's injected by the routing HTTPHeaderModifiers instead of manual header addition from the calling app.

    The service I'm using does have subsets, so maybe if I simplify a bit or try a few other things I can get it to work... But before I dump more time into playing around with it, wanted to check if HTTPHeaderModifier injection for maglev hashing is even possible. Anyone have any ideas? Maybe @blake ? Thanks!
    Mark Martirosian
    @markmartirosian
    Hello, getting "memberlist: Failed to resolve i-032e19c9eecd21e.region/[IPv6 address]: lookup [IPv6 address]: no such host" messages. The cluster is otherwise operational. Any idea how to fix?
    Owen Byrne
    @byrneo

    Hello, when I apply a ProxyDefaults resource on Kubernetes (1.18), this error appears in the consul-controller logs:

    2021-12-01T13:15:21.855Z    INFO    webhooks.proxydefaults    validate create    {"name": "global"}
    2021-12-01T13:15:21.971Z    INFO    controller.proxydefaults    config entry not found in consul    {"request": "default/global"}
    2021-12-01T13:15:21.978Z    INFO    controller.proxydefaults    config entry created    {"request": "default/global", "request-time": "3.458308ms"}
    2021-12-01T13:15:22.004Z    ERROR    controller.proxydefaults    Reconciler error    {"reconciler group": "consul.hashicorp.com", "reconciler kind": "ProxyDefaults", "name": "global", "namespace": "default", "error": "Operation cannot be fulfilled on proxydefaults.consul.hashicorp.com \"global\": the object has been modified; please apply your changes to the latest version and try again"}
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227

    Here's what I'm applying:

    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
    metadata:
      name: global
    spec:
      config:
        envoy_public_listener_json: "{\"@type\":\"type.googleapis.com/envoy.config.listener.v3.Listener\",\"name\":\"public_listener:0.0.0.0:20000\",\"address\":{\"socket_address\":{\"address\":\"0.0.0.0\",\"port_value\":20000}},\"filterChains\":[{\"filters\":[{\"name\":\"envoy.filters.network.http_connection_manager\",\"typed_config\":{\"@type\":\"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\"stat_prefix\":\"public_listener\",\"forward_client_cert_details\":\"APPEND_FORWARD\",\"set_current_client_cert_details\":{\"subject\":true,\"dns\":true,\"uri\":true},\"route_config\":{\"name\":\"public_listener\",\"virtual_hosts\":[{\"name\":\"public_listener\",\"domains\":[\"*\"],\"routes\":[{\"match\":{\"prefix\":\"/\"},\"route\":{\"cluster\":\"local_app\"}}]}]},\"http_filters\":[{\"name\":\"envoy.filters.http.router\"}]}}]}]}"

    Error aside, it does appear to have created the resource ok:

    kubectl get proxydefaults.consul.hashicorp.com
    NAME     SYNCED   LAST SYNCED   AGE
    global   True     17m           17m

    This anything to worry about?

    André Ilhicas
    @Ilhicas
    Folks, I just wanted to come here to say thank you so much for the version 1.11 virtual ip feature
    I was able to use it to have transparent proxy in nomad using default dialed directly as false using virtual ips and it just works great.
    So thanks for all the engineering effort in making this possible
    :love_letter:
    3 replies
    DarkEdges
    @darkedges
    Afternoon, does anybody have any tips on how to debug ingress gateway hosts issues. I am getting a 404 when I use the following
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: IngressGateway
    metadata:
      name: ingress-gateway
    spec:
      tls:
        enabled: true
      listeners:
        - port: 8080
          protocol: http
          services:
            - name: frontend
              hosts: 
              - "frontend.xxx.xxxx.com"
              - "localhost"
    2 replies
    DarkEdges
    @darkedges
    Morning, I am a little confused between ingress gateway and ingress controller for k8s. Seems all the demos use traefik or kong as the ingress controller, but then what is the use of the ingress gateway? Do I need to use both? The reason for asking is that it does not seem possible to have a cert-manager (connected to HashiCorp Vault) to issue a Certificate for the ingress gateway, and that you need to have an ingress controller for that. But If I use something like traefik whats the point of also configuring a gateway with Splitting and Resolver?
    2 replies
    Narendra Patel
    @narendrapatel
    Hi, uinstalled consul following https://www.consul.io/docs/k8s/operations/uninstall but post this deployments are failing. Get the error :
    Internal error occurred: failed calling webhook "consul-connect-injector.consul.hashicorp.com": Post "https://consul-connect-injector-svc.consul.svc:443/mutate?timeout=10s": service "consul-connect-injector-svc" not found
    1 reply
    tirelibirefe
    @tirelibirefe
    Hello Ladies & Gentlemen, I've installed Consul as tls enabled on Kubernetes. I am not able to find the same certificate in any secret which is shown here kubectl exec consul-consul-server-0 -n consul -- curl -sk https://localhost:8501/v1/connect/ca/roots | jq -r .Roots[0].RootCert Could you please advise?
    Narendra Patel
    @narendrapatel

    Hi, needed a bit of clarification around the below,

    The service name registered in Consul will be set to the name of the Kubernetes service associated with the Pod. This can be customized with the consul.hashicorp.com/connect-service annotation. If using ACLs, this name must be the same as the Pod's ServiceAccount name.

    Is it mandatory for the service name to be same as service account or is it only mandatory when we use the annotation consul.hashicorp.com/connect-service?

    2 replies
    Narendra Patel
    @narendrapatel
    In our case we have an app deployed on k8s which consists of multiple pods, all of which use the same service account. How would consul name services in this case? Also since we annotate at pod spec level how does consul identify its associated service since as per docs the consul service name is set to the name of the Kubernetes service associated with the Pod.
    1 reply
    Kholis Respati Agum Gumelar
    @kholisrag
    Hi all, is there a way to advertise public ip in consul helm chart?
    2 replies
    Narendra Patel
    @narendrapatel
    Hi, Is there a way to disable circuit breaking for the local app cluster. Facing some very high latency issues when scaling number of requests.
    local_app::default_priority::max_pending_requests::1024
    local_app::default_priority::max_requests::1024
    local_app::default_priority::max_retries::3
    local_app::high_priority::max_connections::1024
    local_app::high_priority::max_pending_requests::1024
    local_app::high_priority::max_requests::1024
    local_app::high_priority::max_retries::3
    13 replies
    iluciv
    @iluciv:matrix.org
    [m]
    Hey there, I'm hoping someone can clarify where I'm failing here. I simply want to create two tokens so 2 sets of dev teams have access to 2 sets of key values for their apps. We have for instance app-kv key store and 2 folders in that app-1 app-2 I have two policies (policy-dev-team1 and policy-dev-team2) that are simply key_prefix "app-kv/app-1"{ policy="write"} and key_prefix "app-kv/app-2" { policy = "write"} and I've generated two tokens token-dev-team1 and token-dev-team2 attaching to these policies respectively. This approach works with OSS consul v1.9.3 but with the ent v 1.10.4 each token can see the app-kv store but nothing under that. If I change the policy to key_prefix "app-kv" {policy="write"} the tokens can see everything underneath (so both app-1 and app-2 kv's) I know it's not set up correctly how it is (very new to this and this is an inherited setup.) so happy to be schooled on how it should be laid out (for instance should I just create a dev-team-1 and dev-team-2 namespace? I was just looking for a quick segregation of access without having to refactor how these key values are stored/called. Thanks for taking the time to read the ramble
    2 replies
    mntzn
    @mntzn:matrix.org
    [m]
    Hi, been going through https://learn.hashicorp.com/tutorials/consul/docker-compose-auto-config, is it somehow possible to avoid using hostname in `vault write identity/oidc/ .... template='{"consul": {"hostname": "consul-client" } }'? something like one of node_meta keys or even a wildcard?
    Alex Oskotsky
    @aoskotsky-amplify
    Hi. I'm seeing an issue where my services fail the "Connect Sidecar Listening" health check every 10 minutes for a couple of seconds at a time. Has anyone ever seen this happen before? Could it be related to envoy hot restart? @Blake
    11 replies
    This seems to only happen in consul 1.9 and above
    Alex Oskotsky
    @aoskotsky-amplify
    I see in the logs that the "<service name>-sidecar-proxy:1" check is getting synced every few minutes. It seems to get reset into critical state until the check is completed. This didn't happen in earlier versions of consul. Any idea what would cause it to repeatedly sync that check when nothing has changed about it?
    iluciv
    @iluciv:matrix.org
    [m]
    Thanks @blake will read through that. We really need to plan namespaces more thoughtfully I was really just wanting a quick and dirty policy to get the devs going then refactor with a better structure
    iluciv
    @iluciv:matrix.org
    [m]
    Any one know of a chat to talk about the fabio load balancer at all?
    1 reply
    mntzn
    @mntzn:matrix.org
    [m]

    1.11.2 deb package does not ship with empty /etc/consul.d/consul.env anymore which is used by consul.service:

    [Service]
    EnvironmentFile=/etc/consul.d/consul.env
    ...

    this causes consul to fail on startup

    Jan 14 10:47:39 REDACTED systemd[1]: consul.service: Failed to load environment files: No such file or directory
    4 replies
    iluciv
    @iluciv:matrix.org
    [m]
    Hey @blake thanks just wanting more to understand how it's working. We recently spun up a dev ent version hashistack consul, nomad,vault and fabio for nomad conatiner lb /ssl I had made a full chain pem file for the wildcard cert to be used by fabio for the nomad jobs. Fabio is configured as a nomad job (docker conainters) on the linux worker nodes and as a windows service on the windows worker nodes. As the linux fabio are docker containers deployed by nomad they check into consul with a 443 health check and the fabio url webprefix. The windows services have been set up as a script to install the service, the fabio windows service config (fabio.properties file and certs) is on a cifs share and it only checks into consul with an 8080 healthcheck. We had an issue with any windows containers were not getting issued ssl certs for their webprefix url until I added in a key.pem and cert.pem file with the full chain pem file I already had in the certs folder. So that issue got resolved but I don't fully understand whats happening here. As the linux jobs are the only ones with the web prefix and the 443 health check in consul are they the only nodes issuing the certs? Is there something I should have configured to improve this? I really had a hard time finding logs of any vaule on the windows nodes. Should I be setting the cert path to consul or vault instead? At the moment the nomd jobs mounts a volume (nfs) and ingests the certs from there and the windows as I said gets they're config and certs off a cifs share. (fwiw we have 4 windows worker nodes and 4 linux worker nodes currently in this cluster)
    Saurabh Rawat
    @eklavya
    I have:
    • 3 instances with public ips, labelled as separate datacenters.
    • firewall rules to allow tcp/udp on 8300-8302 ports.
    • bind_addr on all 3 nodes as 0.0.0.0
    • network rules are fine since I can ssh to all these nodes from each other over their public ips.
      yet I keep getting no route to host while trying to WAN join these togather, any ideas why?
    Kholis Respati Agum Gumelar
    @kholisrag
    Hi all, using consul-template in nomad job template stanza, can we get metadata of the nodes that consul service reside ? if so how can we do it ?
    2 replies
    Theerapong Kulawong
    @Theerapong

    Hi all,

    I got this error,

    Error creating: Internal error occurred: failed calling webhook "consul-consul-connect-injector.consul.hashicorp.com": failed to call webhook: Post "https://consul-consul-connect-injector-svc.myconsulns.svc:443/mutate?timeout=10s": dial tcp 10.105.230.122:443: i/o timeout

    while I was trying to deploy the Deployment.

    My 3 Consul servers are outside Kubernetes.
    I install Consul clients using Helm.
    (All Consul v1.11.2)

    when I use this command to see logs
    "kubectl -n myconsulns logs -f pod/consul-consul-connect-injector-webhook-deployment-59d4d85bg79sk"

    2022-01-18T05:58:32.603Z        ERROR        controller.endpoints        failed to get service instances        {"name": "consul-consul-connect-injector-svc", "error": "Get \"http://192.168.90.187:8500/v1/agent/services?filter=Meta%5B%22k8s-service-name%22%5D+%3D%3D+%22consul-consul-connect-injector-svc%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22myconsulns%22+and+Meta%5B%22managed-by%22%5D+%3D%3D+%22consul-k8s-endpoints-controller%22\": dial tcp 192.168.90.187:8500: i/o timeout"}
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile

    In Helm, if I set "connectInject.failurePolicy " to be "Fail", I got the above error.
    (If I set "connectInject.failurePolicy " to be "Ignore", I will not get the above error. But there are no SlideCars when I see via "consul catalog services" command)

    Could anyone suggest to me how to solve this issue, please.

    jdaniel-at-yottly-dot-com
    @jdaniel-at-yottly-dot-com
    Hello, I noticed that consul tracks its server instance (server port) among services, nomad for example registers all instances (rpc, serf, http, ...), can I convince consul to register other ports as well for more complete discovery?
    6 replies
    Michael Aldridge
    @the-maldridge
    @blake I just finished reading the roblox postmortem. My hat is off to the consul team for really stepping up and debugging that close to the metal. Going inside the storage primitives really demonstrates how dedicated hashicorp is to building a quality product. Figured you might be able to pass along the positive feedback!
    1 reply
    bbuddha
    @bbuddha
    @blake Can you please look into this when you got a minute. Feeling stuck as this is a critical use case for us to recommend building a service mesh using consul enterprise.
    https://discuss.hashicorp.com/t/setup-envoy-as-ingress-gateway/34547/2
    1 reply
    tretinha
    @tretinha
    Hey, I'm trying to connect two services using Consul Connect, I'm doing a sidecar via Nomad and I'm constantly getting "gRPC config stream closed since connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED". I have TLS enabled on Consul and Nomad, using self-signed certificates. Can someone help me with that? I'm not sure if I'm missing some step from the docs, but I thought Consul would handle the
    Sorry, the message went cut in half, but to complete: I thought Consul would handle the connection between everyone with its built-in CA. That's it :)
    Alex Oskotsky
    @aoskotsky-amplify
    Does consul DNS support looking up a subset defined in a service-resolver for a service?
    3 replies
    Roi Ezra
    @ezraroi
    I am using the consul-k8s project and using it to sync k8s to consul. Looks like that services are synced to consul only on what is called fullSync and on changes. From the comments in the code of the consul-k8s it seems that it should sync changes when they happen and only on full sync (happens every 30 seconds by default). Have anyone else encountered this? is this on purpose?
    Jesse Adelman
    @boldandbusted
    Howdy. Can I use 'consul snapshot restore' on a non-Leader member of a cluster? I just tried it, and I didn't get an error, but it seems like the snapshot wasn't applied?
    2 replies
    (Better jargon: a 'follower' server)
    Jesse Adelman
    @boldandbusted
    (My belief about it not being applied is only from the Index number not rolling back.)