Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Michał Frąckiewicz
    @SystemZ

    Is RBAC any different on OVH k8s cluster than let's say GCP?
    One of the devs from argo Slack said me this:

    I’m afraid I don’t know what could be the problem. You may need to check with OVH why this wouldnt work

    yctn
    @yctn
    @SystemZ no rbac is rbac. but how rbac itself is designed could be very differently yes
    Michał Frąckiewicz
    @SystemZ

    Ok, I'll write more details.
    There is yaml which considering other yamls should give one pod some godlike permissions on cluster:

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-application-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argocd-application-controller
subjects:
- kind: ServiceAccount
  name: argocd-application-controller
  namespace: argocd

    yet, it doesn't have any:

    argocd@argocd-application-controller-5d5866cf56-8lbkd:~$ kubectl get clusterroles
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:argocd:argocd-application-controller" cannot list clusterroles.rbac.authorization.k8s.io at the cluster scope

    Any idea how to debug it?

    Philippe Vienne
    @PhilippeVienne_gitlab
    @SystemZ You are specifying a argocd-application-controller ClusterRole (line 8) isn't cluster-admin you want to designate ?
    Michał Frąckiewicz
    @SystemZ
    If I recall correctly, I tried it to and doesn't work either
    Let me try again...
    Philippe Vienne
    @PhilippeVienne_gitlab
    Edit your cluster role binding then recreate your pod (otherwise secret JWT is not refreshed)
    Michał Frąckiewicz
    @SystemZ
    oh, it needs restart? ok, let's try it
    Christian
    @zeeZ
    Does it really? That'd be an important detail I also didn't know
    Michał Frąckiewicz
    @SystemZ
    I removed pod, it recreated itself, still no enough permissions with cluster-admin
    argocd@argocd-application-controller-5d5866cf56-ct94d:~$ kubectl get clusterroles
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:argocd:argocd-application-controller" cannot list clusterroles.rbac.authorization.k8s.io at the cluster scope
    Christian
    @zeeZ
    kubectl -n argocd get serviceaccount argocd-application-controller
    kubectl describe clusterrole argocd-application-controller
    kubectl describe clusterrolebinding argocd-application-controller
    Those should work and match as a first sanity check, unless I mistyped on mobile
    Also check if that cluster-admin role really exists
    Michał Frąckiewicz
    @SystemZ
    systemz@pc:~$ kubectl -n argocd get serviceaccount argocd-application-controller
NAME                            SECRETS   AGE
argocd-application-controller   1         11h


systemz@pc:~$ kubectl describe clusterrole argocd-application-controller
Name:         argocd-application-controller
Labels:       app.kubernetes.io/component=application-controller
              app.kubernetes.io/name=argocd-application-controller
              app.kubernetes.io/part-of=argocd
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"ap...
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]



systemz@pc:~$ kubectl describe clusterrolebinding argocd-application-controller
Name:         argocd-application-controller
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"argocd-application-controlle...
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name                           Namespace
  ----            ----                           ---------
  ServiceAccount  argocd-application-controller  argocd
    yep, it exists
    systemz@pc:~$ kubectl get clusterrole
NAME                                                                   AGE
admin                                                                  264d
argocd-application-controller                                          11h
argocd-server                                                          11h
calico                                                                 264d
calico-node-3.6.0                                                      173d
cloud-controller-manager                                               264d
cluster-admin                                                          264d
...
    Christian
    @zeeZ
    kubectl auth can-i list clusterroles.rbac.authorization.k8s.io
    As the Argo account. Still doesn't make any sense to me why it wouldn't work
    Michał Frąckiewicz
    @SystemZ
    systemz@pc:~$ kubectl auth can-i list clusterroles.rbac.authorization.k8s.io
Warning: resource 'clusterroles' is not namespace scoped in group 'rbac.authorization.k8s.io'
yes

argocd@argocd-application-controller-5d5866cf56-ct94d:~$ kubectl auth can-i list clusterroles.rbac.authorization.k8s.io
no
    Christian
    @zeeZ
    Which means there is either an absolutely stupid facepalm thing I'm missing, or your cluster is weird
    Michał Frąckiewicz
    @SystemZ
    I'm curious if I can replicate that on fresh OVH k8s cluster
    Christian
    @zeeZ
    Try kubectl auth can-i .... --as=system:serviceaccount:argocd... on your admin account, substitute accordingly
    Michał Frąckiewicz
    @SystemZ
    something like this?
    systemz@pc:~$ kubectl auth can-i list --as=system:serviceaccount:argocd-application-controller clusterroles.rbac.authorization.k8s.io
Warning: resource 'clusterroles' is not namespace scoped in group 'rbac.authorization.k8s.io'
no
    Christian
    @zeeZ
    Yeah. You're missing the namespace after serviceaccount: though
    sys:sa:ns:acc
    Michał Frąckiewicz
    @SystemZ

    oh man, this are long strings in a cmd :)

    kubectl auth can-i list --as=system:serviceaccount:argocd:argocd-application-controller clusterroles.rbac.authorization.k8s.io
Warning: resource 'clusterroles' is not namespace scoped in group 'rbac.authorization.k8s.io'
no

    still "no", though

    Michał Frąckiewicz
    @SystemZ
    Hmmm, I started new 1.11 cluster to replicate my setup and it's "yes"
    so my cluster is misconfigured somehow
    Dennis van der Veeke
    @MrDienns
    is rbac enabled on our clusters? in the kubernetes docs, it says "To enable RBAC, start the apiserver with --authorization-mode=RBAC", which I dont think we end users can do?
    Thomas Coudert
    @thcdrt
    Hello @MrDienns , you can see all features enabled here: https://docs.ovh.com/gb/en/kubernetes/exposed-apis-software-versions-reserved-resources/
    Indeed RBAC are enabled
    Dennis van der Veeke
    @MrDienns
    fantastic, thank you
    if I specify in my k8s role that that role only has access to the secrets in a specific namespace, does that mean that that role will have access to all secrets in that namespace? is there some way (if needed) of specifying that my role only has access to one particular secret, while also being in a shared namespace?
    Christian
    @zeeZ
    Yes. You can specify a resourceNames list in your roles rules
    Dennis van der Veeke
    @MrDienns
    thank you very much, i shall try that out later
    Christian
    @zeeZ
    Official rbac doc has it in an example. Not that it doesn't work for all verbs (or didn't last I checked), such as list
    Dennis van der Veeke
    @MrDienns
    as long as I can prevent a pod from reading any other secret than the one it needs, it's good :)
    Michał Frąckiewicz
    @SystemZ
    How long does adding smallest B2-7 node to k8s cluster can take?
    I think it's now more than 20 mins in "Installing" state
    Thomas Coudert
    @thcdrt
    It should take several minutes
    More than 10 mn it begins to not be normal
    Michał Frąckiewicz
    @SystemZ
    I'll give info in another 10mins, just to be sure
    Thomas Coudert
    @thcdrt
    Can you send me your cluster id in private please ?
    Michał Frąckiewicz
    @SystemZ
    ok
    Michał Frąckiewicz
    @SystemZ
    @thcdrt thx, it's working now :)
    Thomas Coudert
    @thcdrt
    You're welcome, don't hesitate to ping us as you did if it seems to you a bit too long.
    Michał Frąckiewicz
    @SystemZ
    I guess monitoring for creation > X min would be great, then no PM needed from customers ;)
    Thomas Coudert
    @thcdrt
    Yep, that's planned, we have it only on clusters for now
    Guillaume
    @dracorpg
    @thcdrt regarding k8s node deployment delay, I believe the new Public Cloud manager UI is simply updated much more slowly than the actual node state
    I created a node yesterday, it was showing in kubectl get node (and actually starting to schedule pods) about 3~4 min later but the UI was still showing "Installation in progress" like... 10 min later?