Supervisor cluster in configuring state with error on Service: velero.vsphere.vmware.com. Reason: ReconcileFailed.

Products

• VMware vSphere with Tanzu

Issue/Introduction

- We see the Supervisor Cluster shows in configuring state with message:

     Configured Core Supervisor Services
         • Service: velero.vsphere.vmware.com. Reason: ReconcileFailed. Message: Expected to find at least one version, but did not                 (details: all=0 -> after-prereleases-filter=0 -> after-kapp-controller-version-check=0 -> after-constraints-filter=0).
         • Service: tkg.vsphere.vmware.com. Status: Running

- All Supervisor Control Plane nodes show as 'Ready', etcd of the nodes are all healthy, all pods are in running state in the Supervisor.

- Velero shows service endpoint IP
   kubectl get svc -A | grep -i velero

- When listing packages on the Supervisor we do not see velero package listed.
   k get packages.data.packaging.carvel.dev -n vmware-system-supervisor-services | grep -i velero

- Describe shows:

   Status:
     Conditions:
       Message:               Expected to find at least one version, but did not (details: all=0 -> after-prereleases-filter=0 -> after-kapp-controller-version-check=0 -> after-                   constraints-filter=0)
       Status:                True
       Type:                  ReconcileFailed
     Friendly Description:    Reconcile failed: Expected to find at least one version, but did not (details: a...
     Last Attempted Version:  1.6.1-embedded+23741747
    Observed Generation:     6
     Version:                 1.6.1-embedded+23741747
     Events:                    <none>

- In the wcp logs we could see below snippets:

  2024-12-17T09:24:25.745Z error wcp [appplatform/kube_supervisor_service_repo_impl.go:454] [opID=67653899] Error fetching kube            PackageMetadata (vmware-system-supervisor-services/velero.vsphere.vmware.com) on Supervisor e9a63ad0-d047-488f-b345-e8c2095a0559 from    db: sql: no rows in result set Carvel PackageMetadata (namespace: vmware-system-supervisor-services, name: velero.vsphere.vmware.com)    cannot be found, continue to return service info

Environment

vSphere with Tanzu or vSphere Kubernetes Service

Cause

Velero package is missing on the Supervisor cluster.

Resolution

- Verify the Velero package is present on the vCenter:

   ~ ]# ls -la /etc/vmware/wcp/supervisorservices/packages/
   total 36
   drwxr-xr-x 2 root root  4096 Nov 11 09:00 .
   drwxr-xr-x 3 root root  4096 Nov 11 09:00 ..
   -rw-r--r-- 1 root root  7137 Aug 19 20:42 tkg-package.yaml
   -rw-r--r-- 1 root root 19723 Aug 19 20:42 velero-package.yaml

- Restart wcp service on the vCenter.
  service-control --restart wcp