Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Peter Borghard
    @peterborghard
    hey folks, anyone know how to add a header in consul connect? I'm able to match and set the destination. But can't seem to figure out the config for header manipulation.
    3 replies
    Peter Borghard
    @peterborghard
    Anyone know of any good docs on setting up an http2->grpc proxy using consul connect? I can't seem to get it working, tried a few different ways.
    3 replies
    Markus Keil
    @thereapman
    Hi All,
    I'm facing failed_eds_health issues between a sidecar and a service in another data center using Meshgate WAN federation.
    the sidecar sits in a K8s deployment that has its consul servers externally (no consul srvs inside k8s).
    Accessing the service works fine from a non-k8s sidecar on a VM in the same DC as the k8s.
    The Helm chart config reference doesn't give any clues on what i'm missing.
    Is there a way to debug the health check path or any other leads i could follow?
    2 replies
    Blake Covarrubias
    @blake
    Have ideas or need some troubleshooting assistance for Consul? Meet with us and tell us all about it! Sign up at https://hashicorp.sjc1.qualtrics.com/jfe/form/SV_b7cNuNBMrPr4b8q.
    madhucs
    @madhucs:matrix.org
    [m]
    Need help , Trying to run CONSUL on EKS (Kubernetes cluster) , I see pods are not comming up the exception I see for pod is below ->

    2022-04-27T21:40:58.980112134Z
    ==> failed to parse /consul/config/..2022\_04\_27\_21\_29\_57.194001731/server.json: 1 error occurred:
    \* invalid config key auto\_reload\_config


    This is my configuration, deploying through HELM 3+
    client:
    enabled: false
    nodeSelector: |
    dev/group: tools
    tolerations: |
    - key: "tools"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
    global:
    datacenter: adapt
    name: consul
    server:
    enabled: true
    extraConfig: |
    {
    "dns\_config": {
    "service\_ttl": {
    "\*": "15s"
    },
    "node\_ttl": "5s",
    "max\_stale": "5m"
    }
    }
    image: consul:1.10.3
    nodeSelector: |
    adapt/group: tools
    resources:
    limits:
    cpu: 300m
    memory: 300Mi
    requests:
    cpu: 300m
    memory: 300Mi
    tolerations: |
    - key: "tools"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
    ui:
    enabled: true
    service:
    annotations: |
    service.beta.kubernetes.io/aws-load-balancer-name: mynlb
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags:Environment=Dev
    type: LoadBalancer
    3 replies
    madhucs
    @madhucs:matrix.org
    [m]
    Woohoo @blake Consul 1.12.0 worked , thank you very much it saved lots of my time !! :-)
    Blake Covarrubias
    @blake
    You're welcome. :-)
    Alex Oskotsky
    @aoskotsky-amplify
    I am sporadically getting these errors upstream connect error or disconnect/reset before headers. reset reason: connection termination: 0 when using consul connect. Is there a workaround for this? I see that Envoy has an option to retry on reset connections. Is that configurable in Consul?
    18 replies
    Matt Darcy
    @ikonia
    my test lab running 3 consul servers all failed at once (due to a power outage), my understanding would be that when they came back up, they would reform the cluster and elect a leader based on the raft state, however, the 3 nodes have come back up, and they haven’t elected a leader, consul operator raft list-peers shows error 500 no cluster leader, consul operator raft list-peers -stale shows the other 2 nodes in the 3 node cluster (I’m on one of the master) showing as followers
    why are they not coming up and relecting a leader, and there must be a better way to recover this than remove the peers.json and reboot
    one thing to note that I cannot understand is in the list-peers command the nodes all list their correct IP address, apart from one, that shows it’s address as 127.0.0.1:8300 - the config file does not set it to listen on localhost, so I’m not sure why this node is reporting as 127.0.0.1
    Matt Darcy
    @ikonia
    also in 1.12 when I start a master, I get ‘protocol 2 is spoken by default’ - all my estate is 1.12 so why is not using raft protocol v3. ?
    consul operator raft list-peers (cluster now formed) shows protocol version 3, but consul members list shows everyone on protocol 2
    Matt Darcy
    @ikonia
    tidying up my consul lab - it looks like lots of minor errors/config issues/unknows, is there a way to do a ‘consul ping’ from a node to another node, one of my nodes is complaining "transport: Error while dialing dial tcp <nil>->10.11.216.234:8300: operation was canceled” - yet that node is up and running just fine, and I want to try to get some understanding as to why this consul client thinks that node is unavailable
    Matt Darcy
    @ikonia
    oddly after it complains about this it then states Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 which includes one of the nodes it’s complaining about
    Matt Darcy
    @ikonia
    I’m using a hardware air sensor with an exporter written for prometheus ( https://github.com/tijmenvandenbrink/enviroplus_exporter ) seems to be working great, if I curl http://localhost:8000/metrics I get a complete set of metrics
    my prometheus config uses consul with consul_sd_configs to get dynamic service offerings, works great with node_exporter and various others, however on this one, I’ve configued the consul service definition pretty much the same as node_exporter as its dumb, it just queries http://host:port/metrics
    as said above querying the host with curl http://localhost:8000/metrics works fine, but when I look at consul’s service in the consul gui, I can see it’s querying http://localhost:8000/metrics exactly the same as my curl test, but half the metrics are missing each time
    6 replies
    why would consul display partial metrics when curl is showing the exporter is offering out the full set
    and I can see in the healthcheck that consul is correclty using exactly the same url as my curl test
    Matt Darcy
    @ikonia
    thanks to some help from @blake (appreciated) I’ve manged to test and tweak my consul home lab setup and learnt a little bit in the process, I’m offering up some data to prometheus (sucessfully) via consul_sd_configs as a service discovery setup, which again is working. in order to use the data I need to set two labels in the prometheus config ‘group: something’ and ‘location: something’ in my old config, I’d just set prometheus a target and some labels, but I’d assume there is a way to add these labels into a service consul is ‘offering’
    are there any docs that can be suggested that explain how to add labels to a service that consul is offering
    Matt Darcy
    @ikonia
    from what I’m reading it’s not as simple as just using the ‘tag’ option as labels
    Matt Darcy
    @ikonia
    I’m sure I could just use the relable_configs section, and then use source_labels and set a target, but that feels like hard coding the labels for something that is a dynamic service,
    Matt Darcy
    @ikonia
    ahh. maybe I’m missusing tags, should I be using ‘meta’ instead of tags and trying to re-label ?
    Matt Darcy
    @ikonia
    from what I’m reading, tags are the right way to do it, but then I don’t see how I can get those tags into prometheus as labels without hard coding them in the prometheus config
    Matt Darcy
    @ikonia
    I think I see what I’m doing wrong, I’ve missunderstood the labelling approach to prometheus, I’ll revisit how I label, plus just realised I should have been replying in thread, not on seperate lines, I’ll correct that going forward
    Konig-Corey
    @Konig-Corey
    Can someone please tell me how to use the consul cli to pull the current config value for HTTP Maximum Connections per Client- http_max_conns_per_client. Looking to hotfix some servers and id like to confirm the configuration has been taken but I can’t seem to figure out how to list the config for that entry (if at all possible). Thank you in advance
    6 replies
    Konig-Corey
    @Konig-Corey

    Try to run a consul reload -token=<token> and get back a Configuration reload triggered but I don’t see any logs that suggest a reload actually took place, and the value we are attempting to reload http_max_conns_per_client doesn’t appear to actually have been updated.

    Questions:

    • is there a way to get a more verbose output for reload?
    • is there something else that needs to be done to reload this value? the docs suggest this is a reloadable configuration value.
    madhucs
    @madhucs:matrix.org
    [m]
    Question on Consul Authentication -> I understand one of the ways is to set a secret token which is part of consul deploy (ACL). Is there a way to enable auth only for UI Access and not for API access?
    1 reply
    axsuul
    @axsuul:matrix.org
    [m]
    Hi I'm on Ubuntu 22.04 now and from the Consul packages available I'm only seeing 1.8.7, a super old version. Is this the only version available?
    axsuul
    @axsuul:matrix.org
    [m]
    PLease ignore that 😛
    Narendra Patel
    @narendrapatel
    Hi, has anyone tried hot restart with envoy similar to reload feature of haproxy / nginx. We are implementing a large scale deployment of envoy via consul and need this functionality to avoid dropping existing connections. There could be instances where we might need this, for eg: reloading in case of issues. There seems to be little documentation for the same on consul website. As per envoy docs we need to use hot-restarter.py. But the start_envoy.sh file seems to be different than consul connect way of starting envoy. What should be the correct way to accomplish this? We are currently using systemctl to manage envoy. Can we configure some thing there for hot restart?
    3 replies
    Konig-Corey
    @Konig-Corey
    Can anyone tell me how to update the http_max_conns_per_client values and reload consule ?
    George Negoita
    @ngmlabs_twitter
    Hello! Is it possible to update the metadata of a node via API (add or delete a key)? I know I can update the config and reload consul, but I was wondering if there is a better solution. Thank you!
    nahsi (Anatoly Laskaris)
    @nahsi:nahsi.dev
    [m]
    @ngmlabs_twitter I think yes since there is a terraform resource for that https://registry.terraform.io/providers/hashicorp/consul/latest/docs/resources/node
    Remy
    @mario-almeida
    What could be the possible reason for this error?
    agent.server.memberlist.lan: memberlist: Was able to connect to X but other probes failed, network may be misconfigured
    6 replies
    Vadym Vikulin
    @vikulin
    @odysseus654, Hi. I saw your project in github go-udt: https://github.com/odysseus654/go-udt. First, I appreciate your affords. It looks like a great job. Could you enable issue in your repo: then I could add a few bits.
    Iury Fukuda
    @iuryfukuda:matrix.org
    [m]
    Hey, someone can help with a question. please?
    when i try to start mesh gateway
    i had some problem
    in am vm environment
    May 30 17:20:06 r1 consul-mesh-start[42577]: [2022-05-30 17:20:06.836][42577][debug][pool] [source/common/conn_pool/conn_pool_base.cc:443] [C23] client disconnected, failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
    May 30 17:20:06 r1 consul-mesh-start[42577]: [2022-05-30 17:20:06.836][42577][debug][router] [source/common/router/router.cc:1154] [C0][S12085588059115559816] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
    May 30 17:20:06 r1 consul-mesh-start[42577]: [2022-05-30 17:20:06.836][42577][debug][http] [source/common/http/async_client_impl.cc:100] async http request response headers (end_stream=true):
    May 30 17:20:06 r1 consul-mesh-start[42577]: ':status', '200'
    May 30 17:20:06 r1 consul-mesh-start[42577]: 'content-type', 'application/grpc'
    May 30 17:20:06 r1 consul-mesh-start[42577]: 'grpc-status', '14'
    May 30 17:20:06 r1 consul-mesh-start[42577]: 'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED'
    May 30 17:20:06 r1 consul-mesh-start[42577]: [2022-05-30 17:20:06.836][42577][warning][config] [./source/common/config/grpc_stream.h:195] DeltaAggregatedResources gRPC config stream closed since 278s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
    1 reply
    the grpc is configured in server
    and tls seens to be good ( i can use it in browser)
    Iury Fukuda
    @iuryfukuda:matrix.org
    [m]
    thanks, it apparently passed
    now its in =s Error registering service "gateway-primary": Put "https://127.0.0.1:8501/v1/agent/service/register": dial tcp 127.0.0.1:8501: connect: connection refused
    1 reply
    Nicolasrs23
    @Nicolasrs23
    Hi everyone,
    I am working with consul v1.12.0 and kubernetes. install some deploys configure service mesh, so far so good.
    The problem is when I want to communicate with an RDS (external service) TCP health checks don't work; I tried two types of approaches with no results:
    • Registering it together with the node, the service and checks via catalog. (output: timeout)
    • Registering a proxy and linking it with the service (output: connection refused)
      Intentions all allow and security groups ok !! here is the repo https://github.com/Nicolasrs23/Consul_proyect.git.
    Marina Shustova
    @MarinaShustowa
    Hello Everyone,
    I’m looking into how Consul can process “Host” http header instead of destination IP for outgoing http requests.
    In my scenario some requests make it to Consul through proxy, so only "Host" header has information about actual destination.
    Is it possible to configure Consul this way?
    Thanks in advance!
    jaiganeshvazhkudai
    @jaiganeshvazhkudai
    hi everyone.. getting a lot of error messages "[ERR] memberlist: Push/Pull with <host> failed: Node <host> protocol version (2) is incompatible: [1, 0] - incidentally If i try to add a new node (client) to the cluster, it fails repeatedly and all i can see in the failure logs is a version of these messages
    Failed to join IP of server : Node 'different host name' protocol version (2) is incompatible: [1, 0]
    5 replies