These are chat archives for UnderstandLingBV/Tuktu

3rd
Aug 2016
ObjectivePartners
@ObjectivePartners
Aug 03 2016 14:09
All three pixel-servers have been failing their health checks since yesterday morning. Something changed, but we are not aware of changing anything.
Erik Tromp
@ErikTromp
Aug 03 2016 14:34
well the servers are still up, the old URL works fine and so does the new one
this is something in the load balancer
ObjectivePartners
@ObjectivePartners
Aug 03 2016 14:40
Exactly. We didn't change anything
Erik Tromp
@ErikTromp
Aug 03 2016 14:45
but there is nothing wrong either, ie. you should check connectivity/righteousness of the load balancer
ObjectivePartners
@ObjectivePartners
Aug 03 2016 14:58
Our side just works fine. Something went wrong with the healthcheck
We didn't change the healthcheck
Erik Tromp
@ErikTromp
Aug 03 2016 15:00
well Tuktu is working fine, so are the URLs you healthcheck (see for yourself, it all works fine)
so if the load balancer says its not OK, go check there
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:01
Something changed there and we didn't do it
The healthcheck is still directing to: HTTP:80/api/v1
Erik Tromp
@ErikTromp
Aug 03 2016 15:03
well I can't even access the load balancer :p
and that URL works fine, see for yourself
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:03
I tried to change it to the new healthcheck, but this didn't solve the issue
The load balancer has 3 unhealthy instances
Non of the tuktu instances give a healthy response
Erik Tromp
@ErikTromp
Aug 03 2016 15:06
so what do you want us to do? when something outside of our control is saying that something within our control is unhealthy, even though it is healthy?
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:06
I don't know
If that is also my response, we don't have a project
My situation is exactly the same
Erik Tromp
@ErikTromp
Aug 03 2016 15:07
do you have logs of the load balancer?
well yeah you could give that answer, but you do control the load balancer
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:10
blob
It's an amazon loadbalancer
I hardly configured it
blob
Thanks Erik
Let's talk about the solution instead of doing the blame game
Erik Tromp
@ErikTromp
Aug 03 2016 15:14
well I don't know
try restarting it
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:15
I'm trying that
I get a 403
Before I was getting a json
The healthcheck asks for the same url
Erik Tromp
@ErikTromp
Aug 03 2016 15:18
ah okay you use a different URL
well that has to do with nginx' config then
you rebooted the server, right?
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:19
Just one of the servers
Erik Tromp
@ErikTromp
Aug 03 2016 15:19
maybe it reloaded a different config that doesnt allow access to that URL
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:19
Not all three
Erik Tromp
@ErikTromp
Aug 03 2016 15:19
well I would start looking there
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:20
I wouldn't
I did look there and couldn't find any recent file updates
Correction. They only work after login on the htpasswd file
Erik Tromp
@ErikTromp
Aug 03 2016 15:27
well I would suggest you go look into the nginx configuration though, it's definitely there
if you keep having issues with infrastructure, I'd suggest you consider upgrading your package
or outsource it to another party
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:28
It is pretty annoying. I agree
Possible issues was my argument to hire you guys:-P
Erik Tromp
@ErikTromp
Aug 03 2016 15:30
we have SaaS versions too you know
hah yeah, that's also why we are also not an infrastructure company, but SaaS would be an alternative if you want to have us do that part for you
might consider it
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:32
Let's have that talk later. I'm trying to dig deeper
ObjectivePartners
@ObjectivePartners
Aug 03 2016 15:39
blob
Here you can see that it happened yesterday morning. I didn't change anything yesterday morning...
Erik Tromp
@ErikTromp
Aug 03 2016 15:44
Restart nginx, if that doesnt helpt, redo the entry for the server serving the URL you are healthchecking