Supervisor cluster in configuring state with error on Service: velero.vsphere.vmware.com. Reason: ReconcileFailed.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• We see the Supervisor Cluster shows in configuring state with message:

Configured Core Supervisor Services
         • Service: velero.vsphere.vmware.com. Reason: ReconcileFailed. Message: Expected to find at least one version, but did not (details: all=0 -> after-prereleases-filter=0 -> after-kapp-controller-version-check=0 -> after-constraints-filter=0).
         • Service: tkg.vsphere.vmware.com. Status: Running

• All Supervisor Control Plane nodes show as 'Ready', etcd of the nodes are all healthy, all pods are in running state in the Supervisor.
  
  
• Velero shows service endpoint IP

kubectl get svc -A | grep -i velero

• When listing packages on the Supervisor we do not see velero package listed.

k get packages.data.packaging.carvel.dev -n vmware-system-supervisor-services | grep -i velero

• Describe shows error:

Status:

     Conditions:
       Message:               Expected to find at least one version, but did not (details: all=0 -> after-prereleases-filter=0 -> after-kapp-controller-version-check=0 -> after-constraints-filter=0)
       Status:                True
       Type:                  ReconcileFailed
     Friendly Description:    Reconcile failed: Expected to find at least one version, but did not (details: a...
     Last Attempted Version:  1.6.1-embedded+23741747
    Observed Generation:     6
     Version:                 1.6.1-embedded+23741747
     Events:                    <none>

• In the wcp logs we could see below snippets:

YYYY-MM-DDTHH:MM:SS error wcp [appplatform/kube_supervisor_service_repo_impl.go:454] [opID=67653899] Error fetching kube PackageMetadata (vmware-system-supervisor-services/velero.vsphere.vmware.com) on Supervisor ########-####-####-####-############ from    db: sql: no rows in result set Carvel PackageMetadata (namespace: vmware-system-supervisor-services, name: velero.vsphere.vmware.com) cannot be found, continue to return service info.

Environment

vSphere with Tanzu or vSphere Kubernetes Service

Cause

Velero package is missing on the Supervisor cluster.

Resolution

• Verify the Velero package is present on the vCenter:

   ~ ]# ls -la /etc/vmware/wcp/supervisorservices/packages/
   total 36
   -rw-r--r-- 1 root root  7137 <Date> 20:42 tkg-package.yaml
   -rw-r--r-- 1 root root 19723 <Date> 20:42 velero-package.yaml

• Restart wcp service on the vCenter to push the missing velero to Supervisor.

service-control --restart wcp