Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Florian Apolloner
    @apollo13
    @taythebot did you install the cni plugins?
    Tay
    @taythebot
    Yup I did! I'm on WSL2, would that make any difference?
    Florian Apolloner
    @apollo13
    uh ah
    no idea if WSL is supported in any meaningfull way
    Tay
    @taythebot
    Huh the job executes on our cluster, so I guess it's an CNI issue! that's unfortunate, I guess i'll just have to test it out on there
    Florian Apolloner
    @apollo13
    Oh wow, I managed to get prometheus to read envoy metrics like this:
    job "whoami-test" {
      datacenters = ["dc1"]
      namespace   = "infra"
    
      group "whoami" {
        network {
          mode = "bridge"
          port "mesh-metrics" { to = 9102 }
        }
    
        service {
          name = "whoami"
          port = 80
          tags = [
            "traefik.enable=true",
            "traefik.consulcatalog.connect=true",
            "traefik.http.routers.whoami.rule=Host(`whoami.infra.example.com`)"
          ]
    
          connect {
            sidecar_service {
              proxy {
                config {
                  envoy_prometheus_bind_addr = "0.0.0.0:9102"
                }
              }
            }
          }
        }
    
        service {
          name = "whoami"
          port = "mesh-metrics"
          tags = ["mesh-metrics"]
        }
    
        task "whoami" {
          driver = "docker"
          config {
            image = "containous/whoami"
          }
        }
      }
    }
    this way I can easily break out metrics to another instance on the same service and let consul scrape that, the plan output is currently bogus, but it works
    Florian Apolloner
    @apollo13
    mhm the consul metrics proxy hates me
    gives me back a 404
    increasing log to debug and copying out the url it proxies to and all is well
    Florian Apolloner
    @apollo13
    lol the metrics proxy doesn't override the host header, that results in the 404 from the LB
    image.png
    finally :)
    gotta somehow switch the service to http so I get more meaningfull info
    pablo platt
    @pablopla_twitter
    @apollo13 prebuilt images might include the node setup but I don't understand how I should manage external resources like dns records, cloud firewall and external load balancers
    Florian Apolloner
    @apollo13
    @pablopla_twitter can you be more specific? you use whatever you like to manage your dns records or firewall
    nomad is a job scheduler it does not open firewall ports or create dns records
    pablo platt
    @pablopla_twitter
    @apollo13 ok so I don't understand what does all the rest
    Florian Apolloner
    @apollo13
    what do you need to do?
    pablo platt
    @pablopla_twitter
    use autoscaler to add/remove worker nodes (media servers)
    add dns records for the media servers and configure cloud firewall for them
    Florian Apolloner
    @apollo13
    well then you probably have to write your own autscaler plugin or tooling to do that
    pablo platt
    @pablopla_twitter
    what type of plugin? I think autoscaler have several kinds
    Florian Apolloner
    @apollo13
    although depending on your load balancers they might be able to automatically discover new nodes?
    pablo platt
    @pablopla_twitter
    media servers use udp with large number of requests/second so load balancers are not practical
    maybe nomad is just not the tool I need. maybe it's more like gcp autoscaling groups
    maybe nomad is best for when you have existing worker nodes and you need to start and stop docker containers on them
    I'm not using containers
    and I do need to start and stop nodes
    Florian Apolloner
    @apollo13
    Yeah I do have a static set of nodes, @the-maldridge might be able to provide some pointers
    Florian Apolloner
    @apollo13
    image.png
    Finally, http metrics :)
    Johannes Gilger
    @heipei_twitter
    Anyone have any experience bringing remote nomad clients into a network via wireguard? Any gotchas?
    I have a local 10.0.0.x network with nomad server/clients and consul, and I basically want to set up 2-3 additional nomad clients across the world and use wireguard to bring their traffic into the existing cluster
    Michael Aldridge
    @the-maldridge
    so many gotchas
    what are you trying to do?
    gc-ss
    @gc-ss

    Some reading of the raft protocol and leadership timeouts are needed to understand the impact of latencies on leader election and issues with flapping

    It's not sufficient that nodes be able to talk to each other - there are latencies that need to be within bounds as well

    Johannes Gilger
    @heipei_twitter
    I know about the latencies, these nomad nodes would only be clients and thus not participate in raft @gc_ss
    @the-maldridge What I'm trying to do is that I have an existing nomad cluster on internal IPs and I simply want to add 2-3 more nomad clients to it so I can manage them within the same cluster, but I don't dare completely changing the running cluster by moving to public IPv4s. The new clients are running on different providers at different geographical locations.
    Michael Aldridge
    @the-maldridge
    yeah my question is to what end
    gc-ss
    @gc-ss
    I'm guessing @heipei_twitter wants to run a few nomad clients on, say, cheap dedicated Hetzner boxes while he maintains his nomad server cluster on AWS for example.
    That way, he gets HA from his nomad server cluster on AWS while enjoying cheap compute on Hetzner
    Michael Aldridge
    @the-maldridge
    but that's not how this works
    that's not how any HA works
    Johannes Gilger
    @heipei_twitter
    I dont want HA from these external boxes. The boxes are unique sbowflakes, they will only get unqiue service jobs. BUT, i want to manage the jobs using nomad, because what is the alternative? Ansible + systemd units, and then you have two separate ways to manage workloads @the-maldridge
    Florian Apolloner
    @apollo13
    @angrycub hcl2 "templating" takes place locally right? how can I pass a hcl2 variable into a template string?
    oh ${var.xyz} -- that will be fun :)
    Florian Apolloner
    @apollo13
        - set -a
        - env |grep -E "^CI_"|awk '$0="NOMAD_VAR_"$0' > nomad.env
        - env |grep -E "^JOB_"|awk '$0="NOMAD_VAR_"$0' >> nomad.env
        # https://gist.github.com/mihow/9c7f559807069a03e302605691f85572#gistcomment-3625310
        - source <(cat nomad.env | sed -e "s/'/'\\\''/g" -e "s/=\(.*\)/='\1'/g")
        - set +a
        - exit_code=0
        - nomad job plan $JOB_SPEC || $exit_code=$?
        - if [ ${exit_code} -ne 0 ]; then nomad job run $JOB_SPEC; fi
    works for now but not exactly nice :D
    also nomad plan doesn't show colors in gitlab ci :/
    Florian Apolloner
    @apollo13
    now it does hashicorp/nomad#10973 :D
    David Marcin
    @djmarcin
    I'm seeing an issue where a docker container I want to run attempts to resolve it's own hostname. I can't easily change this docker container to not do this. Locally, this is fine -- the docker dns resolver responds with the IP address of the container. On nomad, however, the docker DNS resolver does not return an answer. A few questions 1) does this work for other people (so we've messed up the config somehow) and 2) assuming it's not just our config, is there a way to get docker on nomad to resolve these hostnames?
    27 replies