Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jens Peter Secher
    @jpsecher
    Has anyone had any luck with using a reverse-proxy (fabio, træfik, ...) with nomad & consul connect? I’m trying (https://bitbucket.org/jpsecher/nomad-consul-docker) but I have not really succeded yet
    idrennanvmware
    @idrennanvmware
    @Grauwolf_gitlab - we're not having our applications log directly. We dont want to risk losing logs during an http outage so have apps write to a file, and then use filebeat to ship to logz.io/elk Basically we run a sidecar that ships the apps logs so that the app itself isn't concerned with how their log files are being used.

    @jpsecher - we've been having some issues with Nomad CNI and Consul Connect integration. There's some sparse info here hashicorp/nomad#6689

    Dont know if it's the same but we've had no luck getting consul connect + nomad (native integration) to work. Waiting on CNI update to fix the race conditions which should ship in a week or so

    idrennanvmware
    @idrennanvmware
    @Grauwolf_gitlab - it's also possible to run a sidecar that just collects the STDOUT STDERR and fires that off to some logging system. We have one team internally doing that too and they run a logzio container that listens to that output
    Jens Peter Secher
    @jpsecher
    No @idrennanvmware, I’m running on Ubuntu, CNI works fine for the Dashboard/counter. But I cannot figure out how to get a reverse-proxy to use consul connect to route traffic to other nomad-controlled services. The reverse proxy can only talk to services on the same node
    Michael Aldridge
    @the-maldridge
    @Grauwolf_gitlab no my organization has another place that runs our elk stack
    though I have been looking at logz.io recently
    Michael Aldridge
    @the-maldridge
    @shantanugadgil I'm curious what your long term goals with converge was. The fact that its officially abandonware kind of puts a damper on doing anything substantial with it
    Farhad Shahbazi
    @Grauwolf_gitlab
    @idrennanvmware how would a sidecar be able to collect stdout/stderr?
    Shantanu Gadgil
    @shantanugadgil
    @the-maldridge the goal was to minimize (really really minimize) the footprint needed to do config management on the remote machine.
    The config mgmt problem of creating Consul/Nomad configs and updating the binaries
    (abandonware would be a compromise, yes, but anything else which can achieve the same is also fine)
    anything which doesn't need a large(ish) runtime on the remote side should be ok.
    Farhad Shahbazi
    @Grauwolf_gitlab

    Hm. So I was just about setting the logging section for the docker driver to syslog https://www.nomadproject.io/docs/drivers/docker.html#logging

    So, once I set this, I loose nomad logs in the cli/web ui? At least its looking like that at the moment

    Farhad Shahbazi
    @Grauwolf_gitlab
    OK, as a quick-fix i'm using journald logging instead of syslog. Just to get this setup working. But I still wonder if I would loose the feature of having accessing logs via nomad once I try to integrate this with our graylog setup.
    idrennanvmware
    @idrennanvmware
    @Grauwolf_gitlab - we're using logzio/docker-collector-logs for that
    Shantanu Gadgil
    @shantanugadgil
    @Grauwolf_gitlab you mean nomad logs go to local syslog (rsyslog, or such) and then it is forwarded to central server?
    In a test environment I have used docker's syslog driver directly to a central rsyslog server with no major hiccups.
    I think (I could be wrong) ... "docker syslog driver sends to local rsyslog, sends to central rsyslog" could be better tolerant setup for intermediate network outages between worker nodes and the central log server.
    Soheil Pourbafrani
    @ahoora08
    Can I start separate nomad servers and clients in one physical node? for example 3 servers and 3 clients running all on one node?
    James Rasell
    @jrasell
    @ahoora08 yes, but you'll have to avoid a lot of port bind conflicts
    Soheil Pourbafrani
    @ahoora08
    @jrasell Thanks, I couldn't find the port configuration parameter for the server. Is there any defined parameter to set?
    Grant C
    @morethanunpopular
    I have a general question about Nomad: Is the mbits definition for network in the resource stanza enforced somehow? or is it used for scheduling purposes only?
    stoically
    @stoically
    since network quotas exist in the limit stanza, i'd assume its only about allocation
    Justin Debbink
    @jdebbink
    Anyone out there running Windows clients with the Docker driver successfully using Server 2019 and Nomad 0.10?
    Michael Aldridge
    @the-maldridge
    @morethanunpopular I don't think that's enforced in any released version, but I'd love to be proven wrong
    Grant C
    @morethanunpopular
    kk, thanks!
    idrennanvmware
    @idrennanvmware

    QQ on templates (really just starting to use them now). Is there a way to reload environment variables, or even a task WITHOUT restarting the task? When I do noop, I don't see my file being rendered on template change, signal wants to queue a kill, and restart isn't good for my scenario (I end up in endless loops)

    My scenario in a nutshell - create a file with all the current service (meeting a name criteria) that's up to date in each container. So as a container goes up/down I want to rerender that template each time BUT I do NOT want to restart the container(s) that template is attached to

    idrennanvmware
    @idrennanvmware
    I can work around the problem above, was just wondering if there was a way within nomad to rewrite the file with no restart
    Michael Aldridge
    @the-maldridge
    noop should re-render the template, but depending on how you expose the template it won't update
    if I had to guess, you're running docker and are bind mounting the file somewhere into the container?
    Michael Aldridge
    @the-maldridge
    I have made a thing, maybe other people will find it useful: https://github.com/the-maldridge/yurt-tools/tree/master/cmd/up2date
    stoically
    @stoically
    slowly diving into nomad and consul, currently trying to get them both running in dev mode with docker-compose. works so far but nomad somehow advertises itself with ip "127.0.0.1" to consul and hence the health checks fail. any way to get nomad to advertise the docker network ip instead?
    Maxim Payne
    @maximlepayne_twitter
    hmm, i have a docker job starting over and over again. what's weird is that the docker image disappears right after it shuts down again. cleanup delay should be 3m usually, right?
    stoically
    @stoically
    re my question: hardcoding the ips in the compose file and changing nomads bind_addr helps
    Soheil Pourbafrani
    @ahoora08
    Hi, I have a java jar file that only println some string in a while loop. If I run it as nomad job uisng nomad java driver, how could I see the outputs of the println command?
    Shantanu Gadgil
    @shantanugadgil
    @stongo you can run docker in host mode networking
    Soheil Pourbafrani
    @ahoora08
    When I run a java job, I got the error: rpc error: code = Unknown desc = file /opt/java/bin/java not found under path /opt/nomad/data3/alloc/5860658c-5565-3e5b-3476-7adc983d9872/java-task
    • the /opt/java/bin/java is java path in the client node
    • data3 is the data directory of the client
    • I have java installed on all clients, it's included into the path, too
    Michael Aldridge
    @the-maldridge
    I had one node just absolutely eat it, I am now trying to clean up some consul zombies with the myENA consul zombie hunter, but since the node that had the problem is gone it is failing
    short of manually trying to deregister these services with the CLI (which also isn't working) is there another way to remove them from my cluster?
    Michael Aldridge
    @the-maldridge
    turns out I had a node that thought it was still alive, a force-leave helped
    stoically
    @stoically
    my nomad client runs inside a docker container that's spawned by docker-compose and put into its own network. is there a way to tell the client that the docker driver should put containers into the docker-compose network?
    stoically
    @stoically
    ah, i misunderstood the network_mode configuration. you can put custom network names in there and it works
    Tommy Alatalo
    @tommyalatalo
    Is it possible to set multiple tags to use when cloud auto-joining consul? i.e. all tags should be present on the node to auto-join to. Or is it possible to also set "label" for gce auto-join so that you can first filter on host labels and then on tags. I want to avoid getting all nodes that have the tag 'auto-join' in my gce project
    idrennanvmware
    @idrennanvmware
    @the-maldridge sorry for the delayed response. Yes I am using docker. I've posted the template in the original post and I'm writing an .env file (with env=true) and setting the variable. Noop does not rerender the file or change the variable in the template. Am I doing something daft?
    Edit:. I realized I didn't post the snippet. Will do in a few when I get to work
    idrennanvmware
    @idrennanvmware

    Zookeeper instance

    task "zookeeper" {
      template {
        data = <<EOH
                {{$i := env "NOMAD_ALLOC_INDEX"}}
                {{$ip := env "NOMAD_IP_zk"}}
                GARBAGE={{ $ip }}
                ZOO_SERVER_ID={{$i | parseInt | add 1}}
                ALLOW_ANONYMOUS_LOGIN = "yes"
                ZOO_SERVERS={{range $index, $value := key "core/inventory/zk_server_linux" | split ","}}{{if ne $index 0}},{{end}}{{ $currentIp := $value | replaceAll "[" "" | replaceAll  "]" "" | trimSpace }}{{ if eq $currentIp $ip}}0.0.0.0:2888:3888{{else}}{{$currentIp}}:2888:3888{{end}}{{end}}
    EOH
        destination = "zk-env/zk.env"
        env         = true
        change_mode = "restart"
        splay       = "1s"
      }
    @the-maldridge - I know its set to restart right now, but if I make it noop then I don't see the environment variables changed when a watch is triggered
    Shantanu Gadgil
    @shantanugadgil
    Do the extra spaces makena difference when env==true? For env files I always write everything stuck together on either side of the = sign.
    Michael Aldridge
    @the-maldridge
    ah, I misunderstood. I don't think you can adjust environment variables like that @idrennanvmware I'm fairly confident that the environment is a set-once property