VCF Fleet Lifecycle upgrade fails with Platform Health Check Error [VCFMS-HEALTH-002]

Products

VCF Operations

Issue/Introduction

When attempting to upgrade the VCF Fleet Lifecycle upgrade task fails with the following error:

Platform Health Check Error [platform-vmsp-platform-validate-resources-vsphere-resources : unable to validate vSphere objects: unable to resolve resourcePool identifier ResourcePool:resgroup-##: ServerFaultCode: The object 'vim.ResourcePool:resgroup-##' has already been deleted or has not been completely created, unable to resolve cluster identifier ClusterComputeResource:domain-c##: ServerFaultCode: The object 'vim.ClusterComputeResource:domain-c##' has already been deleted or has not been completely created, unable to resolve network identifier DistributedVirtualPortgroup:dvportgroup-##: ServerFaultCode: The object 'vim.dvs.DistributedVirtualPortgroup:dvportgroup-##' has already been deleted or has not been completely created] [VCFMS-HEALTH-002]

Environment

VCF Operations 9.1
VCF Fleet Lifecycle 9.1
VCF Services Runtime 9.1

Cause

The issue occurs because the compute resources (Cluster, Resource Pool, and Network) were changed for VCF Services Runtime VMs. Because the compute resources were changed, the existing configuration on vmsp became outdated and still pointed to the old, now-invalid objects causing the validation errors during the upgrade.

Resolution

If compute resources are changed in vSphere, the configuration on vmsp needs to be updated to match the new environment. Update the vmsp pd configuration with the new cluster, resource pool, and network information.

Phase 1: Identify the new names or paths or IDs in vCenter

Log in to the vSphere Client and note the exact new inventory names and paths for your recreated infrastructure.

Phase 2: Update the stale configuration in VCF Services Runtime

Connect to the VCF Services Runtime control plane via SSH.
Log in as the vmware-system-user.
Switch to root access by running: sudo -i
Back up the current Provider Definition (pd) configuration by running:
```
k get pd -n vmsp-platform -o yaml > /tmp/pd-backup.yaml
```
Open the configuration for editing:
```
k edit pd -n vmsp-platform
```

Scroll down to the values: provider: vsphere: block. It will look similar to the following, containing the stale path values:

#The following is an example. Make sure to just replace the incorrect values.

values:
      provider:
        vsphere:
        cluster: /path
        clusterId: ClusterComputeResource:<NEW-DOMAIN-ID>
        network: /path/network/<NEW-PORTGROUP>
        networkId: DistributedVirtualPortgroup:<NEW-DVPORTGROUP-ID>
        resourcePool: /path/host/<NEW-CLUSTER>/Resources
        resourcePoolId: ResourcePool:<NEW-RESGROUP-ID>

Modify the cluster, network, and or resourcePool values to match the new friendly names or paths you identified in vCenter during Phase 1.
Save and exit the editor (if using vi/vim, press Esc, type :wq, and press Enter).
The vmsp pods will automatically detect the configuration change and reconcile. This process can take upto 30 to 60 minutes.
Retry the VCF Fleet Lifecycle upgrade in VCF Operations.