vSphere Supervisor Workload Cluster TKC in RoleBindingSyncFailed State due to Undefined Namespace in Rolebinding

Products

• VMware vSphere Kubernetes Service
• Tanzu Kubernetes Runtime

Issue/Introduction

While connected to the Suprvisor clster contxt, the following symptoms are present:

• Describing the workload cluster's TKC shows the following error message, indicating that the namespace value is missing. Values in brackets <> will vary by environment:

  • kubectl describe tkc <cluster name> -n <cluster namespace>

  • Conditions:
         Last  Transition Time: YYY-MM-DDTHH:MM:SSZ
         Message:                    failed to create ClusterRoleBinding: ClusterRoleBinding.rbac.authorization.k8s.io “<role binding name>” is invalid: subjects[0].namespace: Required value
         Reason:                      RoleBindingSyncFailed
         Severity:                      Warning
         Status:                         False
         Type:                           Ready

• When viewing the TKG Controller pod, the following error messages are present for the affected workload cluster, where values in brackets <> will vary by environment:
  •  

    kubectl get pods -A | grep tkg-controller
    
    kubectl logs -n <tkg controller namespace> <tkg-controller name>

  • controller/rolebinding-controller "msg"="Reconciler error" "error"="failed to create ClusterRoleBinding: ClusterRoleBinding.rbac.authorization.k8s.io \"<rolebinding name>\" is invalid: subjects[0].namespace: Required value" "name"="<cluster name>" "namespace"="<cluster namespace>"
    
    rolebinding_controller.go:124] vmware-system-tkg-controller-manager/rolebinding-controller/<cluster namespace>/<cluster name> "msg"="Failed sync/purge RoleBindings while reconciling" "error"="failed to create ClusterRoleBinding: ClusterRoleBinding.rbac.authorization.k8s.io \"<rolebinding name>\" is invalid: subjects[0].namespace: Required value"

 

While connected to the affected Workload Cluster context, the following symptoms are observed:

• Describing the rolebinding noted in the error message shows that the namespace value is missing or empty in the Subjects table at the bottom:

  • kubectl describe rolebinding <rolebinding name> -n <rolebinding namespace>

  • Subjects:
       Kind                Name                    Namespace
       ---                 ---                     ---
       ServiceAccount      <account name>                   

Environment

vSphere with Tanzu 7.0

vSphere with Tanzu 8.0

Cause

This error is due to a user-created rolebinding in a Supervisor cluster namespace, referencing a ClusterRole "Edit" which is used by the tkg-controller to sync ClusterRolebindings to the TKC in the namespace. The rolebinding's subjects has a serviceaccount or user account specified without a namespace.

This causes the TKC to report the error of "RoleBindingSyncFailed". As a result of this error, new users/service accounts added to the vSphere namespace will not be mapped to a clusterrolebinding in the TKC.

 

Resolution

The problematic rolebinding for the affected workload cluster will need to be corrected to include the missing namespace value.

1. Connect to the affected workload cluster context
   
   
2. Locate the problematic rolebinding and confirm that it is missing the namespace field:

   • kubectl get rolebinding -n <namespace>

3. Edit the rolebinding to include the missing namespace value under the Subjects table:

   • kubectl edit rolebinding <rolebinding name> -n <namespace>

   • Subjects:
     
     Kind            Name              Namespace
     ---             ---               ---
     ServiceAccount  <account name>