These are chat archives for FRosner/cluster-broccoli

6th
Nov 2017
John Doe
@tituturumtu_twitter
Nov 06 2017 18:28
Hi. I'm just upgrading the whole cluster (nomad/consul) and have problem with cluster broccoli. It does not connect to Consul. I tried passing the option -Dbroccoli.consul.url but it didn't help. I don't see anything suspicious in the logs - am I missing something?
Frank Rosner
@FRosner
Nov 06 2017 19:32
Was it working before? Did you start any job?
@tituturumtu_twitter The connection is lazy at the moment (that's a bit annoying). So it will say that it's not connected until it actually has to discover some services.
Which version of Broccoli are you using? Which version of Nomad and Consul?
John Doe
@tituturumtu_twitter
Nov 06 2017 19:34
Hi, I'm using the broccoli docker image (v08
nomad: 0.7.0, consul 1.0.0
I just upgraded the whole cluster. My templates were discovered but when I click the start button - nothing happens
Frank Rosner
@FRosner
Nov 06 2017 19:36
And did you also upgrade Broccoli? Or is it a fresh new installation? There are some breaking changes going to 0.7.0 and then 0.8.0 respectively.
John Doe
@tituturumtu_twitter
Nov 06 2017 19:36
I pulled the latest docker image and mounted a docker volume with old templates
Frank Rosner
@FRosner
Nov 06 2017 19:37
Ok that sounds like a problem with Nomad, not Consul. When we upgraded to a newer Nomad version they changed some stuff in the job definition(I remember one issue with the ephemeral disk stanza).
The new Broccoli version changed a few things around (see the links posted above) when it comes to configuration etc.
Also at 0.7.0 the instance format has changed. There is a converter available. What do the logs say when you press the button?
Or even before.
John Doe
@tituturumtu_twitter
Nov 06 2017 19:40
I didn't see any interesting messages in logs
will that help if i mounted an empty directory for instances?
and have just the templates?
Frank Rosner
@FRosner
Nov 06 2017 19:41
Then you for sure don't have to worry about migrating them for now.
I don't know if it will make the error go away.
John Doe
@tituturumtu_twitter
Nov 06 2017 19:43
I have a small three node cluster that was shut down today for maintenance (upgrade) so I don't have to keep the instances description
Frank Rosner
@FRosner
Nov 06 2017 19:43
Usually if a job cannot be send there should at least be something in the log. What log level did you configure? Did you also check the Nomad logs? If Nomad gets a response it doesn't understand it should also log it.
John Doe
@tituturumtu_twitter
Nov 06 2017 19:43
good point
I'm using debug in broccoli
Frank Rosner
@FRosner
Nov 06 2017 19:43
Wow ok so that's gonna be a lot of logs :D but at least there should be something in there.
John Doe
@tituturumtu_twitter
Nov 06 2017 19:44
yep - that was the assumption :)
Frank Rosner
@FRosner
Nov 06 2017 19:44
When you are still stuck you can try to replace the template variables manually (e.g. with Jinja or sed or a text editor) and see if you can send the job JSON to Nomad using curl.
If this succeeds, then there is something wrong with Broccoli.
If not, then it's most likely either your templates not being up to Nomad 0.7.0
John Doe
@tituturumtu_twitter
Nov 06 2017 19:45
ok - I'll try that
Frank Rosner
@FRosner
Nov 06 2017 19:48
Let me know how it goes. And tell me what you find in the Nomad logs when you click the button.
Sorry for the troubles!
John Doe
@tituturumtu_twitter
Nov 06 2017 20:01
there is a problem with the convert script - mktemp complains there is "not enough X's in the parameter" :)
it can be fixed by changing the line:
tmpInstanceFile=$(mktemp -t "$instanceFileName")
to:
tmpInstanceFile=$(mktemp -t "${instanceFileName}.XXXX")
this may be a problem with mktemp version in centos
it also added a "\n" string instead of newline character o_O
had to fix it with sed :)
Frank Rosner
@FRosner
Nov 06 2017 20:06
Yeah the conversion script is really hacky :D
I also had problems with different versions of JQ
Next time I'll write it using a proper language, not bash.
John Doe
@tituturumtu_twitter
Nov 06 2017 20:26
I gave up - will try to recreate instances from templates, but it looks like it is not working
I'm trying to add instance, specify parameters as in old instance file but when I hit apply - nothing happens
Frank Rosner
@FRosner
Nov 06 2017 20:27
what
That all sounds very fishy.
You wanna make a screen sharing session real quick?
Can you scroll up? Error messages are shown at the top as I didn't have time to make them more contextual.
Also still the logs don't say anything when you hit apply?
With scrolling up I mean in the UI.
John Doe
@tituturumtu_twitter
Nov 06 2017 20:30
were there any changes in templates?
I mean, in the format
Frank Rosner
@FRosner
Nov 06 2017 20:31
There is no template format really. What it does is just to replace the variables and then send whatever comes out to Nomad.
We are currently working on replacing the template engine with Jinja to make it more flexible but that PR is still open.
John Doe
@tituturumtu_twitter
Nov 06 2017 20:31
I see there was a change
no meta.json
Frank Rosner
@FRosner
Nov 06 2017 20:31
ahhhhh right
it's now called template.conf
John Doe
@tituturumtu_twitter
Nov 06 2017 20:32
did the format change?
Frank Rosner
@FRosner
Nov 06 2017 20:32
I didn't document it yet as 0.8.0 is not released yet
it's HOCON but JSON is a subset
so you can just rename it
John Doe
@tituturumtu_twitter
Nov 06 2017 20:32
ok - I'll try
btw - I wasn't able to pull image with 0.7
there was just 0.8
Frank Rosner
@FRosner
Nov 06 2017 20:33
aw man
thanks for telling me
I've been super busy in the last couple of months so I didn't have many evenings to spend.
I'll check it out. Do you mind creating a github issue?
Also the way I'm documenting it on the Wiki is suboptimal as I can't really version the documentation. So when I'm working on 0.8.0 I either destroy the doc of 0.7.0 or don't put 0.8.0 out, yet. I gotta move the doc somewhere where I can version it properly
John Doe
@tituturumtu_twitter
Nov 06 2017 20:35
sure I will - but first I have to fix my setup - it's quite urgent :D
Frank Rosner
@FRosner
Nov 06 2017 20:35
As I said, happy to make a screen sharing
(also doing my best via chat)
John Doe
@tituturumtu_twitter
Nov 06 2017 20:35
thanks man
Frank Rosner
@FRosner
Nov 06 2017 20:36
You in US?
John Doe
@tituturumtu_twitter
Nov 06 2017 20:36
nope
Poland :)
Frank Rosner
@FRosner
Nov 06 2017 20:36
Haha then I understand why you want to finish it :D
John Doe
@tituturumtu_twitter
Nov 06 2017 20:36
but working for a company that has a HQ in US
Frank Rosner
@FRosner
Nov 06 2017 20:36
i see
Frank Rosner
@FRosner
Nov 06 2017 20:49
I'll have to go to bed soon, @tituturumtu_twitter. Let me know if I can help you with something right now. Otherwise tomorrow.
John Doe
@tituturumtu_twitter
Nov 06 2017 20:50
I'm trying to start the containers from plain nomad job files
will try to play with broccoli later
so thanks a lot - I'll try to catch you tomorrow
have a good night :)
Frank Rosner
@FRosner
Nov 06 2017 20:51
bye!