by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Michael Aldridge
    @the-maldridge
    but you generally shouldn't be hitting a consul server directly anyway
    Blake Covarrubias
    @blake
    @voiprodrigo Consul allows certain queries to be serviced by followers when using a stale consistency mode. It also allows configuring the TTL for DNS records so that responses can be cached for some time. What specifically are you trying to do?
    Rodrigo Pereira
    @voiprodrigo
    I have set stale DNS option. In my mind it made sense to be able to use a leader record to configure in a reverse proxy. I understand that ideally a proper load-balancer with active checks should handle this.
    Tobias Dahlberg
    @somebadcode
    I'm trying to set up a client on a machine and make it part of the cluster. The problem is that when I register a service using the local client, the servers say "no terminating-gateway or ingress-gateway associated with this gateway: gateway=simple-service". The client reports no errors and the healthcheck is fine. The service never shows up in the cluster either. I haven't found any satisfactory documentation regarding this. I often end up in the Connect documentation which is not what I'm trying to do.
    1 reply
    It's like it expects a service mesh but I only want to register a simple service.
    Rogier Wensink
    @Wensink

    Hello. We are using consul for years now and to full satisfaction. But lately we are encountering an issue which we cannot get solved.
    When changes are done to the master cluster and consul is restarted on all masters at the same time, the slaves (and other nodes too) lose the connection. In code we you a join and a rejoin (although the rejoin is completely ignored once the connection is established).

    consul:1.6.2 -join ${CONSUL_JOIN} -retry-join ${CONSUL_JOIN} -node ${LOCAL_IPV4} -advertise ${LOCAL_IPV4} -client 0.0.0.0 -data-dir /data -encrypt ${CONSUL_KEY} -log-level err

    What is the best way to get all nodes connected to the master cluster again, if these are restarted all at once?

    4 replies
    Shantanu Gadgil
    @shantanugadgil
    @Wensink afaik only retry join should suffice. I am not sure why you use both join and retry-join. That said, if you can make the agents "always restart" (with a delay) it should do the job.
    For systemd based consul agent I keep a short delay (5 or 10 seconds) between service restarts
    (i am talking about the equivalent for docker, in your case)
    If you are on a cloud, you should try out thd cloud auto join ... so the problem of server changing ips also is solved
    Rogier Wensink
    @Wensink

    We have consul running in AWS. For joining I thought it would first try the -join. And when that fails, continue with the -retry-join. But if only -retry-join will do, I'll remove the first part.
    The restart you are talking about is that for the master-cluster? To not restart all at once, but keep a delay in place for that?

    The auto-join we tried yesterday, and works nicely. But we didn't continue that, because it didn't fix our issue

    Tobias Dahlberg
    @somebadcode
    Do you have to set up a terminating or ingress gateway for service registration to fully work?
    When ever I register a service, all I get is:
    "no terminating-gateway or ingress-gateway associated with this gateway: gateway=simple-service"
    Tobias Dahlberg
    @somebadcode
    Registering a service in Consul that's running in dev mode works just fine but I can't get it to work at all when I have a cluster with ACL. The local agent client accepts it but the servers never registers it and just return the error I quoted. :(
    Tobias Dahlberg
    @somebadcode
    Is it mandatory to configure something like nginx, HAProxy or similar for Consul to even accept a service registration?
    Tobias Dahlberg
    @somebadcode
    Seems to be an ACL issue. The service shows up in the UI, the local agent that's in client mode shows it but I can't get a the expected result from asking the servers through DNS or HTTP. Trying to figure out how to make anonymous lookups work.
    Tobias Dahlberg
    @somebadcode
    Okay. It was the ACL causing trouble. I forgot the anonymous token. I've updated it and it's possible to use DNS now. Thanks for being my rubbery ducky while I debugged this.
    glokeshathena
    @glokeshathena

    Team, I have defined global-defaults.hcl with the "proxy-defaults" configuration for envoy_prometheus_bind_addr and tracing configuration.

    And I confirm that the configuration is loaded on to the consul agent with
    consul config read -kind proxy-defaults -name global

    But in the generated bootstrap json for envoy, I don't see the listener for prometheus endpoint and tracing clusters info in the generated bootstrap json.

    consul connect envoy -bootstrap -sidecar-for visit-bill

    Any hint/trouble shooting tip to figure out why consul connect command is not honoring the global configuration while generating the bootstrap config for envoy would be highly appreciated.

    ShellFu
    @shellfu

    Hey all when attempting to delete a config kind service-resolver i receive service "foobar" does not have subset named "v1"

    Although when I curl the config/service-resolver endpoint I see it there

    Can I force the removal some how>
    Blake Covarrubias
    @blake
    Interesting. Which version are you using? Can you share the config from consul config read -kind service-resolver -name <foobar>?
    ShellFu
    @shellfu
    Sure
      {
        "Kind": "service-resolver",
        "Name": "foobar",
        "DefaultSubset": "v1",
        "Subsets": {
          "v1": {
            "Filter": "Service.Meta.version == 1"
          }
        },
        "CreateIndex": 1364976,
        "ModifyIndex": 1364976
      }
    deleting by name either via cli or curl to the API seem to remove it
    Blake Covarrubias
    @blake
    Does, or does not remove it? Which version of Consul are you using?
    ShellFu
    @shellfu
    I've even tried to overwrite it via PUT with a new payload
    does not remove it
    The enemy remains
    1.8.4
    sorry 1.8.3
    ShellFu
    @shellfu
    The effect of a stale entry like this causes consul-connect-envoy-sidecar to not be able to find the upstream
    Blake Covarrubias
    @blake
    Do you have any other config entries which reference this service-resolver, such as a router or splitter?
    ShellFu
    @shellfu
    I sure do
    router
    ok so that error makes sense. The router wouldn't have a v1 if I deleted the resolver
    and deleted....
    "That detective...is the right question. program terminated" - Alfred Lannings Hologram
    Blake Covarrubias
    @blake
    We could do a better job at surfacing dependencies between config entries in the error messages. Our team is looking at improving the UX for managing config entries. I’ll this feedback to them.
    ShellFu
    @shellfu
    Yeah that would be helpfuul
    Dylan Van Assche
    @DylanVanAssche
    I had a consul cluster setup running. The leader of the cluster used a static IP.
    Now I had to change the IP of the leader (updated the static IP config + the consul config)
    The consul server won't function anymore because of this. It can't elect a leader. If I force a join with consul join <LEADER NEW IP> it fails and tries to contact the old IP for joining. I cleared the data storage of consul, but it still seems to happen...
    Blake Covarrubias
    @blake
    @DylanVanAssche Have you tried any of the recovery steps outlined here? https://learn.hashicorp.com/tutorials/consul/recovery-outage
    Dylan Van Assche
    @DylanVanAssche
    @blake Thanks! That helped! I was looking in different places than just the recovery outage tutorial :)
    Jean Prat
    @saez0pub
    Hello, I'm searching a way to remove some envoy headers in my ingress gateway : "server: envoy" and "x-envoy-upstream-service-time: 2"
    theballdredge
    @theballdredge
    is there a default location for consul-template to read a config file on disk?
    something that i wouldnt need to provide as a CLI option?
    Mirza Waqas Ahmed
    @mirzawaqasahmed
    Hi All,
    Any suggestion on compressed base VM images (VMware) that can be used for deploying nomad/consul? Like Alpine VM … with minimal foot print…
    Priyanka Sengupta
    @munali
    What takes precedence? If a consul node has configuration that specifies an agent token e.g. "acl": { "enabled": true, "default_policy": "deny", "enable_token_persistence": true, "tokens": { "agent": "xxx" }} and then consul acl set-agent-token agent yyy...which one takes precedence? If the node is restarted, which agent token will it use?
    6 replies
    Victor Dasari
    @victorbe90_twitter
    hello, Im using Consul, envoy with Jager tracing and have these deployed on VMs. I’ve two nodes that are part of a service in Consul. I currently have Jaeger running in all-in-one mode on one of the virtual machines and dont see any spans from Envoy proxy being collected in Jaeger.
    Shantanu Gadgil
    @shantanugadgil

    hi @Wensink

    I have had the following idea (for some time now) to tackle the scenario that all consul servers suddenly lose connectivity with all the nodes.

    The idea is to keep a small node in the worker pool which runs a Consul Agent in client mode.
    The job of this node to just be part of the pool, but not contribute towards the compute cluster. i.e. workloads SHOULD NOT run on this node.

    A cron (or something equivalent) should keep restarting the Consul Agent every so often (say 5 or 10 minutes).

    Due to the Consul Agent restarting, the "auto join" discovery will re-run, and the agents (clients and servers) will discover each other
    due to this single node re-running its discovery.

    This is just thought, I haven't had time to experiment this scenario though, but as they say ... "it compiles in my mind OK"
    For even more resiliency, you could increase the number of the small nodes which I mentioned above.

    Do let me know if you do get a chance to try this idea out

    tagging @blake how does the above idea sound?
    Priyanka Sengupta
    @munali
    If I have ACLs enabled on server agents via configuration : "acl": { "enabled": true, "default_policy": "deny", "enable_token_persistence": true }, and a client agent comes up with NO acl configuration, I am able to run consul members on the node with the client agent and view members of the datacenter...that does not seem right.
    4 replies