Hi @erulabs , happy new year! After some weeks of inactivity I re-joined my cluster to kubesail. Looks like a few new glitches have appeared. First, after verifying the cluster , the ui throws this error:
TypeError: Cannot read property 'verified' of undefined
at https://kubesail.com/static/js/14.1c471194.chunk.js:2:958416
at Array.map (<anonymous>)
at r.value (https://kubesail.com/static/js/14.1c471194.chunk.js:2:958202)
at qa (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:331079)
at za (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:330872)
at _s (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:366611)
at gu (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:358020)
at mu (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:357945)
at su (https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:354954)
at https://kubesail.com/static/js/15.c1ca2479.chunk.js:2:306325
If I refresh the ui the error goes away and the new cluster looks and works fine.
After I install the ingress-nginx and cert-manager via the ui, I try and deploy an nginx app from the template and I also create a service and ingress from the nginx App -> Network tab. Watching the agent log the agent doesn't appear to notice the new ingress. Next I restart the agent and now the agent picks up the ingress, but then there is also a warning in the log that repeats every 10 seconds or so:
(2021-01-12T05:50:36.314Z) info: kubesail-agent starting! { "version": "0.20.1" }
(2021-01-12T05:50:36.351Z) info: Connecting to Kubernetes API...
(2021-01-12T05:50:37.026Z) warn: Unable to determine Ingress Controller! You can install an ingress controller via the KubeSail.com interface!
(2021-01-12T05:50:37.980Z) info: Registering with KubeSail...
(2021-01-12T05:50:38.574Z) info: Connected to gateway socket! { "KUBESAIL_AGENT_GATEWAY_TARGET": "https://usw1.k8g8.com" }
(2021-01-12T05:50:38.645Z) info: Agent recieved new agent-data from gateway! {
"clusterAddress": "pi43x7k3s.dklinkman.usw1.k8g8.com",
"firewall": {
"pi43x7k3s.dklinkman.usw1.k8g8.com": 1
}
}
(2021-01-12T05:50:38.661Z) info: KubeSail Agent registered and ready! KubeSail support information: {
"clusterAddress": "pi43x7k3s.dklinkman.usw1.k8g8.com",
"agentKey": "a0737f613f7f19fd7bac865bb35fd8d7",
"version": "0.20.1"
}
(2021-01-12T05:50:38.983Z) info: Agent recieved new agent-data from gateway! {
"clusterAddress": "pi43x7k3s.dklinkman.usw1.k8g8.com",
"firewall": {
"pi43x7k3s.dklinkman.usw1.k8g8.com": 1,
"nginx.pi43x7k3s.dklinkman.usw1.k8g8.com": "0.0.0.0/0"
}
}
(2021-01-12T05:50:46.303Z) warn: Received request, but no Ingress controller was found! You can install one on KubeSail.com in your cluster settings page. { "host": "nginx.pi43x7k3s.dklinkman.usw1.k8g8.com" }
(2021-01-12T05:50:47.983Z) warn: Received request, but no Ingress controller was found! You can install one on KubeSail.com in your cluster settings page. { "host": "nginx.pi43x7k3s.dklinkman.usw1.k8g8.com" }
(2021-01-12T05:50:56.634Z) warn: Received request, but no Ingress controller was found! You can install one on KubeSail.com in your cluster settings page. { "host": "nginx.pi43x7k3s.dklinkman.usw1.k8g8.com" }
(2021-01-12T05:51:06.931Z) warn: Received request, but no Ingress controller was found! You can install one on KubeSail.com in your cluster settings page. { "host": "nginx.pi43x7k3s.dklinkman.usw1.k8g8.com" }
and also the TLS certificate doesn't get finalized because the cert manager is stuck in a http 503 error loop. Restarting the agent again doesn't help.
At this point I downgrade the 0.20.1 agent to 0.19.0 and then everything starts to work normally. The pending cert is finalized and I can access the application via the gateway kgk8.com url. Still with the 0.19.0 agent I can deploy other apps and expose them to the Internet. With new apps the agent picks up the ingress right away and the TLS certificate is completed in about 30 seconds.
Looks like a lot of the error have gone away. I still get this in the agent log right after joining the cluster, but the error goes away once the cert-manager is installed
(2021-01-12T15:07:29.017Z) info: kubesail-agent starting! { "version": "0.21.0" }
(2021-01-12T15:07:29.099Z) info: Connecting to Kubernetes API...
(2021-01-12T15:07:30.657Z) warn: Unable to create ClustuerIssuer, cert-manager is probably still starting up. Retrying in 30 seconds. conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": service "cert-manager-webhook" not found {
"code": 500,
"stack": "Error: conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: Post \"https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s\": service \"cert-manager-webhook\" not found\n at /home/node/app/node_modules/kubernetes-client/backends/request/client.js:231:25\n at Request._callback (/home/node/app/node_modules/kubernetes-client/backends/request/client.js:168:14)\n at Request.self.callback (/home/node/app/node_modules/request/request.js:185:22)\n at Request.emit (events.js:315:20)\n at Request.<anonymous> (/home/node/app/node_modules/request/request.js:1154:10)\n at Request.emit (events.js:315:20)\n at IncomingMessage.<anonymous> (/home/node/app/node_modules/request/request.js:1076:12)\n at Object.onceWrapper (events.js:421:28)\n at IncomingMessage.emit (events.js:327:22)\n at endReadableNT (internal/streams/readable.js:1327:12)"
}
(2021-01-12T15:07:30.661Z) info: Registering with KubeSail...
(2021-01-12T15:07:31.422Z) info: Connected to gateway socket! { "KUBESAIL_AGENT_GATEWAY_TARGET": "https://usw1.k8g8.com" }
(2021-01-12T15:08:13.521Z) info: Agent recieved new agent-data from gateway! {
"clusterAddress": "rpi43x7k3s.dklinkman.usw1.k8g8.com",
"firewall": {
"rpi43x7k3s.dklinkman.usw1.k8g8.com": 1
}
}
(2021-01-12T15:08:13.533Z) info: KubeSail Agent registered and ready! KubeSail support information: {
"clusterAddress": "rpi43x7k3s.dklinkman.usw1.k8g8.com",
"agentKey": "a5878b6019bb003adde6ef2864d8c397",
"version": "0.21.0"
}
(2021-01-12T15:08:31.606Z) warn: Unable to create ClustuerIssuer, cert-manager is probably still starting up. Retrying in 30 seconds. conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": service "cert-manager-webhook" not found {
"code": 500,
"stack": "Error: conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: Post \"https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s\": service \"cert-manager-webhook\" not found\n at /home/node/app/node_modules/kubernetes-client/backends/request/client.js:231:25\n at Request._callback (/home/node/app/node_modules/kubernetes-client/backends/request/client.js:168:14)\n at Request.self.callback (/home/node/app/node_modules/request/request.js:185:22)\n at Request.emit (events.js:315:20)\n at Request.<anonymous> (/home/node/app/node_modules/request/request.js:1154:10)\n at Request.emit (events.js:315:20)\n at IncomingMessage.<anonymous> (/home/node/app/node_modules/request/request.js:1076:12)\n at Object.onceWrapper (events.js:421:28)\n at IncomingMessage.emit (events.js:327:22)\n at endReadableNT (internal/streams/readable.js:1327:12)"
I still have to restart the agent to pick up a new ingress, but now I have to restart for every new ingress. With 0.19.0 I only had to restart the agent once for the first ingress, and the rest would be picked up automatically
The connection to the server localhost:8080 was refused - did you specify the right host or port?
KUBECONFIG
environment variable should point to the configuration file for your kubernetes setup. The configuration file location will vary depending on what kubernetes you are using. The docs should be able to tell you. On my k3s system the file I need is /etc/rancher/k3s/k3s.yaml. Mind the file permissions also. I usually copy that file to ${HOME}/.kube/config. Don't know why--habit I guess.
kubesail/agent:v0.22.0
if you'd like to give that a shot (it's not yet published as stable on the kubesail website). It should resolve all the needing-to-restart issues for ingresses :)
So it seems to be working ok. The ingress is created and the TLS certificate is finalized and the k8g8.com URL works. But there doesn't seem to be any logging now. At least not like before. With 0.19.0 when a new ingress is created the agent would log and new firewall listing with the new host added. Also when the agent was restarted it would log the current firewall list of hosts. but now just:
(2021-01-12T22:11:44.529Z) info: kubesail-agent starting! { "version": "0.22.0" }
(2021-01-12T22:11:46.403Z) info: KubeSail Agent registered and ready! KubeSail support information: {
"clusterAddress": "n40x7k8s.dklinkman.usw1.k8g8.com",
"agentKey": "6e69954ddcd8d642be27b08405e2aabd",
"version": "0.22.0"
}
Functionally it seems to be working fine, just the logging looks different
kubectl create...
). Let me know if you have any questions!
dns.k8g8.com
- this is a dynamic DNS address that will point at your machines public IP address! Very useful for hosting games, SSH servers, and anything non-http that the Gateway tunneling service doesn't address for you. Let me know if you have any questions! Thanks everyone!
1000 monkeys and 1000 typewriters... Didn't really know (nor still know) what I was doing when I tried to connect my RPi to KubeSail and ended up with some persistent errors that I am trying to get rid of so I can start over. Any idea how to clear these red warning boxes every time I log in?
Permission denied from Games
games.xxxxxx.usw1.k8g8.com / Deployment Invalid clusterConfig from db - no credentials found for cluster: games.xxxxxx.usw1.k8g8.com
Unable to fetch namespaces from games.xxxxxx.usw1.k8g8.com
games.xxxxxx.usw1.k8g8.com might be offline?
@/all New features this week! Dynamic DNS, plain-Dockerfile support in Repo-Builder, lots of improvements and bug-fixes around the site, and lots more coming soon! Also excited to announce the launch of https://www.beeperhq.com - a customer of ours built on KubeSail's platform. Thanks everyone and have a great week!
Wow Beeper looks amazing and can't believe its self hosted on a cluster somewhere. Super cool thanks for sharing this gem!
Hey @flynmoose ! Thanks for reaching out - I sent you an email about that error - we're trying to nail it down and eliminate it but its a hard one to recreate! If you go to https://kubesail.com/clusters and click "Settings", you should be able to delete the cluster from the dashboard entirely. Once that's done, you can try re-installing the agent (add-cluster), and it should do the trick.
Emailed you back with some screenshots and a commandline history of my buffonery. Hope it helps.