Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    jflamy-dev
    @jflamy-dev
    Works. Thks much!
    Seandon Mooy
    @erulabs
    No problemo - as always, your feedback drove an improvement for everyone! :heart: thank you!
    jflamy-dev
    @jflamy-dev
    Next up: k3s running on WSL2. Runs fine normally. Trying to add kubesail-agent yields 0/1 nodes are available: 1 node(s) were unschedulable. Was trying k3s as a dry run prior to tackling Digital Oceab because the memory footprint is much smaller than running a kubernetes cluster on Digital Ocean (they actually suggest running the baby versions for single-node clusters).
    jflamy-dev
    @jflamy-dev
    Nothing special in the logs. Comes with Traefik. Since I cannot connect, can't do the kubesail ingress magic.
    Retch
    @Retch
    Hello, i wanted to add my microk8s, and i get this warning: ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding Is this important?
    jflamy-dev
    @jflamy-dev
    @erulabs regarding the node unschedulable issue on k3s: I am running a single node, just running the k3s server. Maybe there is a taint or something similar that is preventing the agent from running ? My intent is to have something as simple as possible to run on a a small DigitalOcean node. I want to be able to publish a recipe whereby a moderately technical person can open a DO account, install k3s, expose it via Kubesail, and either deploy a template. You know, sort of like the shared clusters (sad face). Cheap is the operative word, the target is penny-pinching amateur sport federations and clubs.
    @erulabs I would try microk8s instead but I was so happy to ditch VirtualBox that I won't reinstall it just for that. WSL2 doesn't support snap.
    Seandon Mooy
    @erulabs
    @Retch that warning can be ignored - yoyre just using a very shiney new version of Kubernetes and its warning about some upcoming changes for us to deal with. I did see some errors last night from your agent registration process tho - did you manage to get things working?
    @jflamy-dev hrmm - you might try “wsl —shutdown” and then restarting k3s - i typically recommend docker-desktop on windows for now though. K3s on windows is pretty experimental currently
    DavidCamelo
    @DavidCamelo
    hi @erulabs my cluster went down due to an OOM problem, I killed all the pods from the remote console but my node has not yet recovered.
    image.png
    image.png
    Seandon Mooy
    @erulabs
    Hey @DavidCamelo - Just me just a few moments and I'll look into that for you. Looks like a java process is the culprit :(
    DavidCamelo
    @DavidCamelo
    Yes, I increased the memory usage of my java deployment and ran a job on it and that caused the memory error.
    My fault :(
    Seandon Mooy
    @erulabs
    Ah yes, looks like around 1pm the memory usage spiked up a bit. No worries, we're here to help :)
    DavidCamelo
    @DavidCamelo
    Thanks!
    Seandon Mooy
    @erulabs
    Hey @DavidCamelo - should be back online now - I see the agent is connected. Let me know if you spot any other issues :heart:
    By the way, if we dont reply quickly here, feel free to email support@kubesail.com - we monitor that more closely than this chat
    DavidCamelo
    @DavidCamelo
    Thank you very much, I see that everything is up
    i will decrease the memory usage of my java deployment
    jflamy-dev
    @jflamy-dev
    @erulabs trying to run microk8s on a digital ocean ubuntu. I added the agent, but the connection somehow broke. I currently get
    root@owlcms-tor1-01:~# microk8s kubectl logs  kubesail-agent-758c4485f7-4rp7j -n kubesail-agent
    (2021-01-29T22:52:03.384Z) info: kubesail-agent starting! { "version": "0.23.3" }
    (2021-01-29T22:52:03.786Z) warn: We'll route traffic to this Node's HostPort 80 & 443, because it appears you have a hostPort nginx ingress controller enabled!
    (2021-01-29T22:52:04.198Z) error: KubeSail agentKey and agentSecret rejected! Please re-install this agent at https://kubesail.com/clusters { "status": 404 }
    I had deleted the cluster, expecting that if the agent came knocking back it would reset things and re-register the cluster.
    How does one "reinstall the agent". If I re-run the yaml, I get all sorts of errors about secrets being already there and suchlike.
    Seandon Mooy
    @erulabs
    Ah, if you "force" delete the cluster from the UI, it does break the agent a bit. Go ahead and do a kubectl delete namespace kubesail-agent and then try re-adding it.
    Ideally the UI tells you to delete the agent installation first, but we've made it easier to "force delete" now
    jflamy-dev
    @jflamy-dev
    That did it. Now tackling the connection to my custom domain.
    jflamy-dev
    @jflamy-dev
    @erulabs gave up on microk8s on a 2GB Digital Ocean basic droplet. k3s seems less invasive. I installed k3s without traefik, and then used the kubesail interface to install nginx and cert-manager.
    I get some strange 404 errors after adding my two ingresses (I have two webapps in the cluster).
    pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="officials.jflamy.dev" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-6j7r9" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="owlcms-ingress-1581768128-2354231461-1002489940" "resource_namespace"="default" "type"="http-01" 
    service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="officials.jflamy.dev" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-57tw9" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="owlcms-ingress-1581768128-2354231461-1002489940" "resource_namespace"="default" "type"="http-01" 
    ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="officials.jflamy.dev" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-sm8gv" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="owlcms-ingress-1581768128-2354231461-1002489940" "resource_namespace"="default" "type"="http-01" 
    sync.go:185] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="officials.jflamy.dev" "resource_kind"="Challenge" "resource_name"="owlcms-ingress-1581768128-2354231461-1002489940" "resource_namespace"="default" "type"="http-01"
    Jean-François Lamy
    @jflamy
    @erulabs after fixing the cname which was broken and not pointing to the correct cluster, I now get 503 errors instead of 404 when a challenge is made.
    Seandon Mooy
    @erulabs
    Ack :/ I wonder if restarting the agent again does the trick... If you can browse to the domain and get your app (and not a 503) thats a good indication its not the agent. If you get a 503 then the cert will not be able to work. The certs use the same domain name as your app but with a /.well-known sort of URL path.
    If restarting the agent -does- work, thats a sort of urgent issue I'll try to fix asap... We actually have a couple patches in the pipeline for the agent that might improve that.
    Carl J. Mosca
    @carljmosca
    how can we clean up what appear to be “stale” domains? I have created/deleted/recreated (with a new name) a cluster and noticed that the old dynamic domain is hanging around - is there a way for me to delete it?
    specifically: k3s1.carljmosca.dns.k8g8.com
    Seandon Mooy
    @erulabs
    Ah - thats good idea - those should really be deleted when a cluster is deleted... our Dynamic DNS domains are pretty new
    ill clean that up for you and add some code to delete those when a cluster is deleted
    Carl J. Mosca
    @carljmosca
    Excellent. Thank you.
    jflamy-dev
    @jflamy-dev

    @erulabs .BYOC cluster on a DigitalOcean standard droplet. Installed latest version of k3s, without traefik. Using KubeSail to install nginx, cert-manager, and define ingresses (since you've figured out all the issuer magic and suchlike).

    NAME                        CLASS    HOSTS                  ADDRESS          PORTS     AGE
    publicresults               <none>   public.jflamy.dev      142.93.154.114   80, 443   89s
    cm-acme-http-solver-xb74s   <none>   public.jflamy.dev      142.93.154.114   80        84s
    owlcms                      <none>   officials.jflamy.dev                    80, 443   25s
    cm-acme-http-solver-jmstz   <none>   officials.jflamy.dev                    80        23s

    If I describe the solvers, the http paths are indeed correct for the challenges, and there are annotations to whitelist any possible source. What is peculiar is that I get a 503 error when attempting to reach the challenge. Normally the longest path takes precedence, so the challenges should go first. Is there a particular way to tell my two main ingresses to NOT listen on port 80 ? Any other idea as to why the error would occur? The problem is the same whether or not I use A records directly to the cluster or a CNAME through the kubesail tunnel.

    Dan Pastusek
    @PastuDan
    @jflamy-dev those conflicts shouldn't be a problem, as you said, the most specific path takes precedence. But I don't think the nginx we install works with A records pointed directly to the cluster. That requires nginx listening on a NodePort (this creates conflicts with Managed Digital Ocean clusters, I assume it also does with K3s)
    go ahead and point your domains back to the kubesail CNAME, and I'll check some config settings on our end
    Dan Pastusek
    @PastuDan
    Can you try deleting and recreating the ingresses? I don't see those host names pointed to your cluster on our end. Sometimes if they were created before the kubesail agent was installed then it can get into a weird state
    Also, feel free to add me (pastudan) to your cluster, I would be happy to dig in a bit
    jflamy-dev
    @jflamy-dev
    @PastuDan I'm at wits end. I've deleted the cluster. Recreated it using the built-in names, trying to make sure that works. Now I get something slightly different.
    Log below is the bottom part of kubectl describe challenge followed by the ingresses
    Status:
      Presented:   true
      Processing:  true
      Reason:      Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://officials.owlcms.jflamy-dev.dns.k8g8.com/.well-known/acme-challenge/XGo8MW96CGKL4NSOxxlk356C7v9kjfRbUBEcERxDiXw': Get "http://officials.owlcms.jflamy-dev.dns.k8g8.com/.well-known/acme-challenge/XGo8MW96CGKL4NSOxxlk356C7v9kjfRbUBEcERxDiXw": dial tcp 142.93.154.114:80: connect: connection refused
      State:       pending
    Events:
      Type    Reason     Age   From          Message
      ----    ------     ----  ----          -------
      Normal  Started    25s   cert-manager  Challenge scheduled for processing
      Normal  Presented  23s   cert-manager  Presented challenge using http-01 challenge mechanism
    root@owlcms-tor1-01:~# kubectl get ingress
    Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
    NAME                        CLASS    HOSTS                                      ADDRESS          PORTS     AGE
    publicresults               <none>   results.owlcms.jflamy-dev.dns.k8g8.com     142.93.154.114   80, 443   22m
    owlcms                      <none>   officials.owlcms.jflamy-dev.dns.k8g8.com   142.93.154.114   80, 443   21m
    cm-acme-http-solver-5tz6q   <none>   officials.owlcms.jflamy-dev.dns.k8g8.com   142.93.154.114   80        77s
    cm-acme-http-solver-dmvxh   <none>   results.owlcms.jflamy-dev.dns.k8g8.com     142.93.154.114   80        79s
    the funny part if you scroll to the end of the challenge part is 142.93.154.114:80: connect: connection refused
    Dan Pastusek
    @PastuDan
    ah yes, the dns.k8g8.com addresses won't work unless you have a NodePort as mentioned above. Can you select one of the domains ending in usw1.k8g8.com?
    that ensures they are proxied through our gateway
    jflamy-dev
    @jflamy-dev
    @PastuDan using the built-in usw1 address seems to work, indeed. Do you have an example of a nginx with a node port? Am running k3s straight on the vm, direct behind a firewall. I will read up on the concept -- had only used a nodeport it to provide direct access to a non-standard port before, would have thought that was what nginx was all about in a baremetal scenario...
    Seandon Mooy
    @erulabs
    @jflamy-dev - Most of the time nginx ingress controller will bind a port on the host system automatically, but it depends (there are plenty of options there). The "dns.k8g8.com" address is a "dynamic DNS" address, so it will point at the IP address of your firewall. You can forward a port on your router to make that work (ie: 142.93.154.114:80). The advantage of the gateway addresses is that we tunnel the traffic to the agent directly, so you can avoid all the messy bits about port-forwarding. We're going to make that more clear in the UI soon to indicate which domain is "tunneled" and which one would require port-forwarding (ie: points at your public IP instead of our gateway/agent system). You're right that the host-port is more or less the "bare-metal" option, which is in Kubernetes terms the same as saying "not using a cloud provider". You can checkout a node-port example here: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
    A home-cluster is, in Kubernetes terms, "bare-metal" - that term is a bit promlematic tho - but essentially just means "I dont have a magic cloud-load balancer that Kubernetes knows how to talk to and program automatically"
    jflamy-dev
    @jflamy-dev
    @erulabs I read that nginx+nodeport example, and I haven't quite figured it out yet. 142.93.154.114 is the public address of my Digital Ocean VM. There is a DO-provided firewall that blocks the traffic except for ports 22,80, and 443, so it's not a port forwarding issue.
    What reinforces my confusion is https://sysadmins.co.za/https-using-letsencrypt-and-traefik-with-k3s/ which makes no reference to a load balancer sitting between the firewall and the nodes. It seems that traefik does things differently, and would actually be simpler for me.