Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    3nprob
    @3nprob
    I don't seem to be able to find any settings in Consul to add additional CAs, or separate CAs for WAN.
    Is there any way to get this working or is there a requirement that server TLS certs for all servers in a WAN gossip pool come from the same CA?
    (consul members on server in new DC is yielding 403 ACL not found but I guess that is expected until replication is successful and that the cert issues are preventing replication)
    3nprob
    @3nprob
    This verifies fine on both ends: openssl s_client -showcerts -verify 5 -connect server.$OTHER_DC.consul:8300 < /dev/null (using /etc/hosts to map hostname to server in the other DC)
    3nprob
    @3nprob
    Stll, I get on server in old, primary DC
    [ERROR] agent.server.rpc: failed to read byte: conn=from=$PROXY_IP :32139 error="remote error: tls: bad certificate"
    and on new :
    [WARN]  agent.server.replication.acl.role: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL roles: rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"
    [ERROR] agent.server.connect: error performing intention migration in secondary datacenter, will retry: routine="intention config entry migration" error="rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"
    [ERROR] agent.server.rpc: RPC failed to server in DC: server=$IP:8300 datacenter=$PRIMARY_DC method=ConfigEntry.ListAll error="rpc error getting client: failed to get conn: x509: certificate signed by unknown authority"
    3nprob
    @3nprob
    Probably unrelated but running consul acl bootstrap on new server return Failed ACL bootstrapping: Unexpected response code: 500 (ACL support disabled) (ACL enabled in config , only hardcoded token is replication as per docs)
    3nprob
    @3nprob
    OK, that turned out to be way easier than expected. Had missed the config option ca_path, changing to that and hoisting CAs in there works fine. No need to involve the system trust chain
    Miguel Araujo
    @Maikuh

    Hello. I'm trying to use Consul with Kubernetes (minikube). I'm trying to use the CRDs for Service Intentions, yet when I apply them, I get the following error

    failed calling webhook "mutate-serviceintentions.consul.hashicorp.com": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block

    I followed this tutorial and I get the error both with L4 and L7. Note that via the UI and the API it works, just the CRDs don't.

    Miguel Araujo
    @Maikuh

    It seems that the above happens mostly if all consul services are not ready yet (probably because the CRDs are actually just making HTTP calls to Consul's API, which would make sense). This is an issue since I'm using tools such as Tilt and they create the CRDs at the same time it installs the helm charts. Basically, I'd only be able to make it work if I install the CRDs manually through the terminal.

    Is this a limitation of Consul's CRDs implementation? I've used CRDs from other solutions before (Gloo IIRC) and have been able to install them at the same time as other helm charts and resources, using Tilt, with no issues.

    pablopla
    @pablopla
    Do I have to use ingress gateway to interact with clients outside of the consul cluster?
    or can I just expose a public IP?
    1 reply
    Michael Aldridge
    @the-maldridge
    can anyone on a relatively recent ubuntu confirm that https://learn.hashicorp.com/tutorials/consul/dns-forwarding#systemd-resolved-setup works?
    I can only reliably get it to give me the following error:
    Sep 15 20:22:46 ip-10-2-26-216 systemd-resolved[19974]: /etc/systemd/resolved.conf.d/consul.conf:1: Assignment outside of section. Ignoring.
    Sep 15 20:22:46 ip-10-2-26-216 systemd-resolved[19974]: /etc/systemd/resolved.conf.d/consul.conf:2: Assignment outside of section. Ignoring.
    19 replies
    Mike Cardwell
    @mike.cardwell:grepular.com
    [m]
    Does a consul client need to be in the same datacenter as a consul server?
    2 replies
    lokesp11
    @lokesp11

    We have Consul Cluster on VM's and we have agent deployed on VM's and K8's both:
    Its been working fine but recently we saw an issue.

    Due to OS upgrade one of server was down and all the vm's got in sync with current server peers but somehow agent deployed on k8s was still trying to connect to same server which was down for patching. There was some delay we saw in getting latest state of servers on k8s.

    2021-09-10T08:03:59.183Z [ERROR] agent.client: RPC failed to server: method=KVS.List server=**.**.**.**:8300 error="rpc error making call: rpc error getting client: failed to get conn: dial tcp <nil>->**.**.**.**:8300: connect: connection refused"

    Is there any setting in helm chart which can help in immediate sync and avoid this issue?

    sstent
    @sstent
    @lokesp11 what is your retry-join pointed at? either a list of nodes to try, load balanced IP, or consul dns entry would be good
    lokesp11
    @lokesp11
    @sstent It's list of nodes . Is there any limitation with list of nodes?
    Even if one server is down all server peers/agents will have the info about it leaving the mesh and catalog will also be updated accordingly . I am trying to understand once this info is updated is there any possibility if consul agent will still try to connect the server which not reachable?
    johnny101
    @johnny101:matrix.org
    [m]
    We seem to be hitting an issue with token rotation and the consul-agent token in our Nomad cluster. If indeed an issue that needs fixing (rather than some kind of approach change), I'm not sure if this would fall more under Nomad or Consul. Hence the cross post here in addition to the Nomad room. I described the setup and the issue here: https://github.com/hashicorp/nomad/issues/9813#issuecomment-930456285. If anyone has any feedback, that would be appreciated.
    Ross Has No Clever Friends
    @rosshasnocleve1_twitter
    Hashicorp devs, do any of you know someone who works on go-discover? There is a pull request lingering since February that looks approved but not merged, and a lot of people are waiting for it.
    Blake Covarrubias
    @blake
    The Consul team has been reviewing go-discover PRs pretty regularly. Which PR are you referring to?
    Michael Aldridge
    @the-maldridge
    reviewing yes, but merging relatively infrequently. I'd bet this is in reference to either proxmox or hetzner, as those are the only two from February
    Ross Has No Clever Friends
    @rosshasnocleve1_twitter
    HI Blake, this one: hashicorp/go-discover#167
    Blake Covarrubias
    @blake
    I was mobile earlier and was having some trouble replying from my phone. I tried to send a few messages, but it looks like they didn't go through.
    Michael Aldridge
    @the-maldridge
    the gitter/matrix ecosystem has been a little wobbly today, probably a little bit of both
    Blake Covarrubias
    @blake
    I'll discuss this PR with our team tomorrow, and will ask that someone follow up on the GitHub issue.
    lokesp11
    @lokesp11

    Does consul support CA Signed certificate for tls communication and can it be integrated with vault to get certificate from vault pki? We are exploring option to use VAULT PKI Infrastructure and trying to implement consul tls communication with certificate generated by vault pki instead of inbuild consul CA?

    Please suggest or help in pointing me to similar use case if exist?

    Thanks in advance

    1 reply
    Ross Has No Clever Friends
    @rosshasnocleve1_twitter
    Thanks Blake!
    Yann Huissoud
    @aiqency
    Any updates on hashicorp/consul#8687. Can we access the specific key modified through consul watch keyprefix as of now?
    2 replies
    Narendra Patel
    @narendrapatel
    Hi, has anyone tried the Escape-Hatch Overrides option? Need to configure envoy access logging. Able to configure it directly on envoy but struggling to set it via Consul. Also how are envoy cluster names formed? I see consul configuring something like api.default.dc1.internal.af617b02-1e21-52c2-d297-36b92be86af9.consul. Not sure what does this hexadecimal string signifies.
    5 replies
    John Spencer
    @johnnyplaydrums
    Noob question about service to service communication when services are using random, dynamically allocated ports (like in a Consul/Nomad cluster). If the port is know ahead of time, e.g. port 80, serviceA can talk to serviceB using the dns record serviceB.service.consul:80. But if the port is dynamically allocated, how does serviceA communicate with serviceB? It could find the port via dig SRV or consul API, but that's additional application code and overhead. Is there a better way?
    Yoan Blanc
    @greut
    @johnnyplaydrums you need some kind of load balancer, e.g. https://www.consul.io/docs/connect
    2 replies
    Sam Lee
    @D2Engine_twitter
    hi, i'd like to access consul ui by ingress subpath.
    deployed "hashicorp/consul:1.10.3" helm chart and tried this additional extraConfig. But it's redirected to "/" path . Is there anything i missed ?
      extraConfig: |
        {
          "ui_config": {
            "content_path": "/sandbox/consul"
          }
        }
    1 reply
    Ross Rochford
    @RossRochford_twitter

    @johnnyplaydrums I suspect a lot of people end up using consul upstreams (sidecars) in Nomad simply for the convenience of it. In this scenario Nomad gives your tasks a single addr/port to communicate through.

    It would be nice to have a similarly convenient setup but with the option of bypassing the security features (encryption, intentions) of consul connect. i.e. just the load balancing and service discovery pieces.

    1 reply
    Andreas Anderssson
    @dinapappor
    I am having a problem where consul isn't being updated fast enough with pods in kubernetes.
    Not sure how to solve that problem.
    We have consul-write-interval set to 1s.
    Yet, our loadbalancer (traefik) is still trying to send traffic to old pods.
    I am not sure if it's because
    1) We don't execute healthchecks from consul to kubernetes pods.
    2) The data in consul is stale.
    How have other people solved this problem?
    Narendra Patel
    @narendrapatel
    Hi, is it possible to set mTLS for ingress-gateways? We want to set inbound mTLS to the mesh for one of our services. Not able to find options for the same. Saw an option for mTLS with https://www.hashicorp.com/blog/proxy-ingress-to-consul-service-mesh but it involves too many moving parts. Preference is for Ingress Gateway with envoy proxy.
    4 replies
    Alex Oskotsky
    @aoskotsky-amplify
    I'm seeing a weird issue where every 10 minutes envoy logs a line this this add 3 cluster(s), remove 2 cluster(s). During that time I'll also see /failed_eds_health for a few seconds when viewing the cluster from the envoy admin UI. The nodes show as healthy in the consul UI during time this is happening. This started happening after upgrading to Consul 1.10. Has anyone ever seen this or have any suggestions?
    2 replies
    Chris Johnson
    @chrisjohnson
    @blake in your demo of using transparent proxies, you were JUST about to start typing a curl command so I could finally understand how they decide the service to route to, but then you got sidetracked chasing down problems in the demo hahaha
    Demo of using transparent proxies on VMs, not k8s, that is
    Anyway, it's still unclear to me. What does an application have to be configured to use as the DNS name when making the request? Should the application use http or https? And does the port need to be something specific?
    2 replies
    Right now I'm in the middle of reconfiguring my tooling to produce a local IP and whatever the upstream port is for that service. Using transparent proxies, would it be using the consul DNS name? And would that be the <svc>.service.consul form or <svc>.ingress.consul form?
    J. Gavin Ray
    @jgavinray
    Hello all!
    krishnaprateek6
    @krishnaprateek6
    Hi all, qq? failed to setup alloc: pre-run hook "network" failed: failed to create network for alloc: Failed to pull gcr.io/google_containers/pause-amd64:3.1: API error (500): Get https://gcr.io/v2/: net/http: TLS handshake timeout we are seeing this when we start a nomad job even though we have pause-amd64 image loaded locally but since one of our env's has strictly no internet access to outside world is there a way in nomad that you can force it to not look into google container registry?
    1 reply
    linuxbsdfreak
    @linuxbsdfreak

    Hi . I installed consul on K8s with the following command

    helm -n consul-server install --create-namespace -g hashicorp/consul -f consul-values.yaml

    cat consul-values.yaml                                                                                                                   ✔  10:52:28 
    ---
    
    global:
      enabled: true
      name: consul
      acls:
        manageSystemACLs: true
      metrics:
        enabled: true
        enableAgentMetrics: true
      image: "hashicorp/consul:1.10.3"
      imageK8S: "hashicorp/consul-k8s-control-plane:0.36.0"
    
    prometheus:
      enabled: true
    
    server:
      replicas: 1
    
    client:
      enabled: true
    
    connectInject:
      enabled: false
      transparentProxy:
        defaultEnabled: false
    
    ui:
      enabled: true
      service:
        type: LoadBalancer
    
    controller:
      enabled: true

    I opened the Ingress endpoint. However where do i find the token to login to save data under the KV? i always get 403 since i am not logged in

    1 reply
    Yann Huissoud
    @aiqency
    From a golang perspective what's the benefit of using hcl:"foo" instead of json:"foo"?
    Yann Huissoud
    @aiqency
    basically we are writing our own library and was wondering what is the benefit of using hcl?
    1 reply
    Narendra Patel
    @narendrapatel
    What is the standard way to override envoy configs via envoy_public_listener_json and envoy_listener_json in kubernetes. On VM able to override via sidecar_service block.