nomad/Lobby

nomadproject.io

Jens Peter Secher

@jpsecher

Has anyone had any luck with using a reverse-proxy (fabio, træfik, ...) with nomad & consul connect? I’m trying (https://bitbucket.org/jpsecher/nomad-consul-docker) but I have not really succeded yet

idrennanvmware

@idrennanvmware

@Grauwolf_gitlab - we're not having our applications log directly. We dont want to risk losing logs during an http outage so have apps write to a file, and then use filebeat to ship to logz.io/elk Basically we run a sidecar that ships the apps logs so that the app itself isn't concerned with how their log files are being used.

@jpsecher - we've been having some issues with Nomad CNI and Consul Connect integration. There's some sparse info here hashicorp/nomad#6689

Dont know if it's the same but we've had no luck getting consul connect + nomad (native integration) to work. Waiting on CNI update to fix the race conditions which should ship in a week or so

idrennanvmware

@idrennanvmware

@Grauwolf_gitlab - it's also possible to run a sidecar that just collects the STDOUT STDERR and fires that off to some logging system. We have one team internally doing that too and they run a logzio container that listens to that output

Jens Peter Secher

@jpsecher

No @idrennanvmware, I’m running on Ubuntu, CNI works fine for the Dashboard/counter. But I cannot figure out how to get a reverse-proxy to use consul connect to route traffic to other nomad-controlled services. The reverse proxy can only talk to services on the same node

Michael Aldridge

@the-maldridge

@Grauwolf_gitlab no my organization has another place that runs our elk stack

though I have been looking at logz.io recently

Michael Aldridge

@the-maldridge

@shantanugadgil I'm curious what your long term goals with converge was. The fact that its officially abandonware kind of puts a damper on doing anything substantial with it

Farhad Shahbazi

@Grauwolf_gitlab

@idrennanvmware how would a sidecar be able to collect stdout/stderr?

Shantanu Gadgil

@shantanugadgil

@the-maldridge the goal was to minimize (really really minimize) the footprint needed to do config management on the remote machine.

The config mgmt problem of creating Consul/Nomad configs and updating the binaries

(abandonware would be a compromise, yes, but anything else which can achieve the same is also fine)

anything which doesn't need a large(ish) runtime on the remote side should be ok.

Farhad Shahbazi

@Grauwolf_gitlab

Hm. So I was just about setting the logging section for the docker driver to syslog https://www.nomadproject.io/docs/drivers/docker.html#logging

So, once I set this, I loose nomad logs in the cli/web ui? At least its looking like that at the moment

Farhad Shahbazi

@Grauwolf_gitlab

OK, as a quick-fix i'm using journald logging instead of syslog. Just to get this setup working. But I still wonder if I would loose the feature of having accessing logs via nomad once I try to integrate this with our graylog setup.

idrennanvmware

@idrennanvmware

@Grauwolf_gitlab - we're using logzio/docker-collector-logs for that

Shantanu Gadgil

@shantanugadgil

@Grauwolf_gitlab you mean nomad logs go to local syslog (rsyslog, or such) and then it is forwarded to central server?

In a test environment I have used docker's syslog driver directly to a central rsyslog server with no major hiccups.

I think (I could be wrong) ... "docker syslog driver sends to local rsyslog, sends to central rsyslog" could be better tolerant setup for intermediate network outages between worker nodes and the central log server.

Soheil Pourbafrani

@ahoora08

Can I start separate nomad servers and clients in one physical node? for example 3 servers and 3 clients running all on one node?

James Rasell

@jrasell

@ahoora08 yes, but you'll have to avoid a lot of port bind conflicts

Soheil Pourbafrani

@ahoora08

@jrasell Thanks, I couldn't find the port configuration parameter for the server. Is there any defined parameter to set?

Grant C

@morethanunpopular

I have a general question about Nomad: Is the mbits definition for network in the resource stanza enforced somehow? or is it used for scheduling purposes only?

stoically

@stoically

since network quotas exist in the limit stanza, i'd assume its only about allocation

Justin Debbink

@jdebbink

Anyone out there running Windows clients with the Docker driver successfully using Server 2019 and Nomad 0.10?

Michael Aldridge

@the-maldridge

@morethanunpopular I don't think that's enforced in any released version, but I'd love to be proven wrong

Grant C

@morethanunpopular

kk, thanks!

idrennanvmware

@idrennanvmware

QQ on templates (really just starting to use them now). Is there a way to reload environment variables, or even a task WITHOUT restarting the task? When I do noop, I don't see my file being rendered on template change, signal wants to queue a kill, and restart isn't good for my scenario (I end up in endless loops)

My scenario in a nutshell - create a file with all the current service (meeting a name criteria) that's up to date in each container. So as a container goes up/down I want to rerender that template each time BUT I do NOT want to restart the container(s) that template is attached to

idrennanvmware

@idrennanvmware

I can work around the problem above, was just wondering if there was a way within nomad to rewrite the file with no restart

Michael Aldridge

@the-maldridge

noop should re-render the template, but depending on how you expose the template it won't update

if I had to guess, you're running docker and are bind mounting the file somewhere into the container?

Michael Aldridge

@the-maldridge

I have made a thing, maybe other people will find it useful: https://github.com/the-maldridge/yurt-tools/tree/master/cmd/up2date

stoically

@stoically

slowly diving into nomad and consul, currently trying to get them both running in dev mode with docker-compose. works so far but nomad somehow advertises itself with ip "127.0.0.1" to consul and hence the health checks fail. any way to get nomad to advertise the docker network ip instead?

Maxim Payne

@maximlepayne_twitter

hmm, i have a docker job starting over and over again. what's weird is that the docker image disappears right after it shuts down again. cleanup delay should be 3m usually, right?

stoically

@stoically

re my question: hardcoding the ips in the compose file and changing nomads bind_addr helps

Soheil Pourbafrani

@ahoora08

Hi, I have a java jar file that only println some string in a while loop. If I run it as nomad job uisng nomad java driver, how could I see the outputs of the println command?

Shantanu Gadgil

@shantanugadgil

@stongo you can run docker in host mode networking

Soheil Pourbafrani

@ahoora08

When I run a java job, I got the error: rpc error: code = Unknown desc = file /opt/java/bin/java not found under path /opt/nomad/data3/alloc/5860658c-5565-3e5b-3476-7adc983d9872/java-task

the /opt/java/bin/java is java path in the client node
data3 is the data directory of the client
I have java installed on all clients, it's included into the path, too

Michael Aldridge

@the-maldridge

I had one node just absolutely eat it, I am now trying to clean up some consul zombies with the myENA consul zombie hunter, but since the node that had the problem is gone it is failing

short of manually trying to deregister these services with the CLI (which also isn't working) is there another way to remove them from my cluster?

Michael Aldridge

@the-maldridge

turns out I had a node that thought it was still alive, a force-leave helped

stoically

@stoically

my nomad client runs inside a docker container that's spawned by docker-compose and put into its own network. is there a way to tell the client that the docker driver should put containers into the docker-compose network?

stoically

@stoically

ah, i misunderstood the network_mode configuration. you can put custom network names in there and it works

Tommy Alatalo

@tommyalatalo

Is it possible to set multiple tags to use when cloud auto-joining consul? i.e. all tags should be present on the node to auto-join to. Or is it possible to also set "label" for gce auto-join so that you can first filter on host labels and then on tags. I want to avoid getting all nodes that have the tag 'auto-join' in my gce project

idrennanvmware

@idrennanvmware

@the-maldridge sorry for the delayed response. Yes I am using docker. I've posted the template in the original post and I'm writing an .env file (with env=true) and setting the variable. Noop does not rerender the file or change the variable in the template. Am I doing something daft?

Edit:. I realized I didn't post the snippet. Will do in a few when I get to work

idrennanvmware

@idrennanvmware

Zookeeper instance

task "zookeeper" {
  template {
    data = <<EOH
            {{$i := env "NOMAD_ALLOC_INDEX"}}
            {{$ip := env "NOMAD_IP_zk"}}
            GARBAGE={{ $ip }}
            ZOO_SERVER_ID={{$i | parseInt | add 1}}
            ALLOW_ANONYMOUS_LOGIN = "yes"
            ZOO_SERVERS={{range $index, $value := key "core/inventory/zk_server_linux" | split ","}}{{if ne $index 0}},{{end}}{{ $currentIp := $value | replaceAll "[" "" | replaceAll  "]" "" | trimSpace }}{{ if eq $currentIp $ip}}0.0.0.0:2888:3888{{else}}{{$currentIp}}:2888:3888{{end}}{{end}}

EOH

    destination = "zk-env/zk.env"
    env         = true
    change_mode = "restart"
    splay       = "1s"
  }

@the-maldridge - I know its set to restart right now, but if I make it noop then I don't see the environment variables changed when a watch is triggered

Shantanu Gadgil

@shantanugadgil

Do the extra spaces makena difference when env==true? For env files I always write everything stuck together on either side of the = sign.

Michael Aldridge

@the-maldridge

ah, I misunderstood. I don't think you can adjust environment variables like that @idrennanvmware I'm fairly confident that the environment is a set-once property

Where communities thrive

People

Repo info

Activity

Zookeeper instance