Core Supervisor Service Error - Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed 'Preparing kapp: Getting service account: serviceaccounts "sa-cci-ns-###" not found'

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Supervisor stuck in a configuring state

Initialized vSphere resources
Deployed Control Plane VMs
Configured Control Plane VMs
Configured Core Supervisor Services
 • Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message: Preparing kapp: Getting service account: serviceaccounts "sa-tkg-####" not found.
 • Service: velero.vsphere.vmware.com. Status: Running
Configured Load Balancer fronting the kubernetes API Server

 

From the supervisor context get package install state shows one or more in a failed stated

kubectl get pkgi -n vmware-system-supervisor-services 
vmware-system-supervisor-services   svc-cci-ns.vmware.com                     cci-ns.vmware.com                                    9.1.0-embedded+739b5075   Reconcile failed: Error (see .status.usefulErrorMessage for details)   7d
vmware-system-supervisor-services   svc-dr-operator.vmware.com                dr-operator.vmware.com                               9.1.0-25369307            Reconcile failed: Error (see .status.usefulErrorMessage for details)   5d2h
vmware-system-supervisor-services   svc-metrics-aggregator.vmware.com         metrics-aggregator.vmware.com                        0.1.0                     Reconcile failed: Error (see .status.usefulErrorMessage for details)   6d23h
vmware-system-supervisor-services   svc-tkg.vsphere.vmware.com                tkg.vsphere.vmware.com                               3.6.1-embedded+v1.35      Reconcile failed: Error (see .status.usefulErrorMessage for details)   7d

 

Describe on the pkgi shows the error due a missing service account. 

kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

  serviceAccountName: sa-tkg-8###
  values:
  -...
status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Error (see .status.usefulErrorMessage for
    details)'
  lastAttemptedVersion: 3.6.1-embedded+v1.35
  observedGeneration: 1
  usefulErrorMessage: 'Preparing kapp: Getting service account: serviceaccounts "sa-tkg-8r2##"
    not found'

Checking accounts on the supervisor shows a different account name 

kubectl get serviceinstallconfigs  -n vmware-system-supervisor-services supervisor-service-install-config -o yaml |  grep 'serviceAccount: sa' 

serviceAccount: sa-tkg-8lb##

 

Environment

VCF 9.x

Cause

The ServiceInstallConfig has rotated the service account but the packageinstall is using the old ones. 

Resolution

First get correct service account information from ServiceInstallConfig.

k get serviceinstallconfigs.appplatform.wcp.vmware.com -n vmware-system-supervisor-services supervisor-service-install-config -o yaml

serviceAccount: sa-tkg-8lb##

2] Update each package install that has incorrect service account associated with the service ID using VKS pkgi as example
    Account sa-tkg-8r2## changed to sa-tkg-8lb##

kubectl edit pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

spec:
  packageRef:
    refName: tkg.vsphere.vmware.com
    versionSelection:
      constraints: 3.6.1-embedded+v1.35
  serviceAccountName: sa-tkg-8r2##  <===== this should be same as one from ServiceInstallConfig
  values:
  - secretRef:

3] wait for some time, packages will then go in to reconciled state. Similar to below. 
k get pkgi -n vmware-system-supervisor-services 
NAME                            PACKAGE NAME                            PACKAGE VERSION           DESCRIPTION           AGE    PAUSED
kube-state-metrics              kube-state-metrics.vsphere.vmware.com   2.14.0-25195109           Reconcile succeeded   5d7h   
svc-cci-ns.vmware.com           cci-ns.vmware.com                       9.1.0-embedded+a5f7c5e3   Reconcile succeeded   5d7h   
svc-tkg.vsphere.vmware.com      tkg.vsphere.vmware.com                  3.6.1-embedded+v1.35      Reconcile succeeded   5d7h   
svc-velero.vsphere.vmware.com   velero.vsphere.vmware.com               1.9.0-embedded+25158515   Reconcile succeeded   5d7h