nomad/Lobby

nomadproject.io

xeroc

@xeroc:matrix.org

        "Binds": [
            "/var/nomad/alloc/85600575-5f59-1b46-73e6-81e10a8300ab/alloc:/alloc:z",
            "/var/nomad/alloc/85600575-5f59-1b46-73e6-81e10a8300ab/bash/local:/local:z",
            "/var/nomad/alloc/85600575-5f59-1b46-73e6-81e10a8300ab/bash/secrets:/secrets:z"
        ],

I don't need to chown .. the image I am using (bitnami images) do that

and i would like to avoid creating my own images

Florian Apolloner

@apollo13

uhm why is there a ":z" at the end?

xeroc

@xeroc:matrix.org

[m]

I think that was from a tutorial on the nomad client .. let me check

Florian Apolloner

@apollo13

:z is for selinux, you shouldn't need that on ubuntu

xeroc

@xeroc:matrix.org

[m]

ok , getting rid of it

Florian Apolloner

@apollo13

and I do not see it on debian either

fwiw I doubt that this will fix your problem, but…

xeroc

@xeroc:matrix.org

[m]

agreed

Florian Apolloner

@apollo13

root@8b2f0a45f0e8:/alloc# ls -lisa
total 20
804715 4 drwxrwxrwx 5 nobody nogroup 4096 Jun 13 09:05 .
805112 4 drwxr-xr-x 1 root   root    4096 Jun 13 09:05 ..
804718 4 drwxrwxrwx 2 nobody nogroup 4096 Jun 13 09:05 data
804716 4 drwxrwxrwx 2 nobody nogroup 4096 Jun 13 09:05 logs
804717 4 drwxrwxrwx 2 nobody nogroup 4096 Jun 13 09:05 tmp
root@8b2f0a45f0e8:/alloc# chown 1001:1001 data/
root@8b2f0a45f0e8:/alloc# ls -lisa
total 20
804715 4 drwxrwxrwx 5 nobody nogroup 4096 Jun 13 09:05 .
805112 4 drwxr-xr-x 1 root   root    4096 Jun 13 09:05 ..
804718 4 drwxrwxrwx 2   1001    1001 4096 Jun 13 09:05 data
804716 4 drwxrwxrwx 2 nobody nogroup 4096 Jun 13 09:05 logs
804717 4 drwxrwxrwx 2 nobody nogroup 4096 Jun 13 09:05 tmp

this did just work for me

not an empheral disk, but the normal /alloc, though I do not think there should be any diff

xeroc

@xeroc:matrix.org

[m]

Exactly .. that is the weird part ..

without ephemeral disk, it all works ..

Florian Apolloner

@apollo13

after all empheral just means nomad will copy the stuff around, docker doesn't know anything

xeroc

@xeroc:matrix.org

[m]

if you run my demo from gist and remove the ephemeral disk, it will work

let me write some other job to illustrate what might be the root issue

weird .. i touched a file and it is owned by root.. with and without ephemeral disk

missed the volume bind .. second

Florian Apolloner

@apollo13

@xeroc:matrix.org your demo job has exec, I am not touching that :)

I have no idea how the bind mounts for exec work

xeroc

@xeroc:matrix.org

[m]

sure. I'll do more researching

Florian Apolloner

@apollo13

I will retry my example with an empheral disk though

mhm nope I can (in a docker task) chown an empheral disk with no issues

xeroc

@xeroc:matrix.org

[m]

did you chown /alloc straight away? i mount bind it to somewhere else and chown there,aybe thats the problem

can u share you job file?

Florian Apolloner

@apollo13

job "test" {
  datacenters = ["dc1"]

  group "web" {
    network {
      mode = "bridge"
      port "mirrorlist" {}
    }

    ephemeral_disk {
      size    = "150"
      sticky  = true
      migrate = false
    }

    task "test" {
      driver = "docker"

      config {
        image = "ubuntu:latest"
        command = "sleep"
        args = ["infinity"]
      }
    }
  }
}

Well yes bind mounting it might be a problem

xeroc

@xeroc:matrix.org

[m]

great .. i think that givese a good hint

Florian Apolloner

@apollo13

but you never said that :þ

xeroc

@xeroc:matrix.org

[m]

thanks for the chat, i'll take it from here asap

Florian Apolloner

@apollo13

https://stackoverflow.com/a/50325808

should help

although it is kinda weird, /alloc is also bind mounted

so dunno

maybe also compare the Mounts section from docker inspect and not just Binds

Charlie Voiselle

@angrycub

So I tried to reproduce your issue yestedrday and my artifact downloaded into that docker_entrypoint.d folder fine

Florian Apolloner

@apollo13

wasn't my issue, but I think they also got it to download but didn't manage to get it executable

@angrycub btw did you have any time to look into my TaskKind / ConsulUsage question?

Charlie Voiselle

@angrycub

I need to hand that thread off to someone here, and I loast my last bit of our notes on where you id’d the place that seems like we are dropping them. If you happen to have that context handy, I can pass that over to Seth.

Florian Apolloner

@apollo13

@angrycub Ah sorry to hear; https://github.com/hashicorp/nomad/blob/main/nomad/structs/consul.go#L56-L88 should imo publish https://github.com/hashicorp/nomad/blob/77f6ecbbbffd9306abc05afe3e273c061145411f/nomad/structs/consul.go#L37 and not just Services/KV. The code that should then verify the Kinds is at https://github.com/hashicorp/nomad/blob/77f6ecbbbffd9306abc05afe3e273c061145411f/nomad/consul.go#L254-L264

Charlie Voiselle

@angrycub

Thanks! I’ll add that to my notes and talk to him about that.

Florian Apolloner

@apollo13

as it stands now I think usage.Kinds will always be empty -- I think that summarizes it :)

he should feel free to contact me if needed ;)

Charlie Voiselle

@angrycub

10-4. I’ve got it captured in that gist that I’d written up earlier. Thanks for helping me rehydrate that context.

Florian Apolloner

@apollo13

uh, great commit hashicorp/nomad@2a640f0

Florian Apolloner

@apollo13

I wonder how many bind mounts a system can handle :D

xeroc

@xeroc:matrix.org

[m]

mhm nope I can (in a docker task) chown an empheral disk with no issues

I was able to fix my shit using a prestart task with raw_exec to chown the NOMAD_ALLOC_DIR. Thanks for pointing me to the right direction.

scyd

@scyd

Hi all, I am currently using Nomad task group and raw_exec to replace my systemd and use it to define dependencies between tasks. Do you know a way to disable Nomad force killing the whole allocation when one task fails or exit? Since I want to debug cause of the issue I am preventing restarts by setting restarts {attempts=0}. Thanks for your help.

Where communities thrive

People

Repo info

Activity