Error: "vmsp-platform: wrong resource state: In Progress VCFMS-HEALTH-002", during VIDB upgrade failure to 9.1

Products

• VMware Cloud Foundation
• VCF Operations

Issue/Introduction

• VIDB upgrade fails during Import Source Component Configuration Pre-check.
• The UI displays the following error messages:

  Running Component Import Prechecks. Status: Failed

Platform Health Check Error [platform-snapshots-snapshot : error listing VMs: the server was unable to return a response in the time allotted, but may still be processing the request (get vspherevms.infrastructure.cluster.x-k8s.io)] [VCFMS- HEALTH-002]

Platform Health Check Error [platform-package-core : 1 of 1 resources are not ok: vmsp-platform: wrong resource state: In Progress - package deployment is in progress:] [VCFMS-HEALTH-002]

  Platform Health Check Error [platform-networking-daemonsets-core : 1 of 1 resources are not ok: istio-cni-node: wrong resource state: InProgress - Available:3/4;] [VCFMS-HEALTH-002]

  
  
  
  
• VMSP logs contain these entries:
  
  vcf-management-services/vcf-fleet-lcm/vcf-svc-<id>/vcf-fleet-upgrade-service-fleetupgrade-########-####/fleetupgrade/console-log-#######.log

{timestamp=YYYY-MM-DDTHH:MM:SS.###Z, level=ERROR, message={id=com.broadcom.vcf.lcm.upgrade.VmspTaskFailed, defaultMessage=Platform Health Check Error [platform-snapshots-snapshot : error listing VMs: the server was unable to return a response in the time allotted, but may still be processing the request (get vspherevms.infrastructure.cluster.x-k8s.io)] [VCFMS-HEALTH-002], localizedMessage=Platform Health Check Error [platform-snapshots-snapshot : error listing VMs: the server was unable to return a response in the time allotted, but may still be processing the request (get vspherevms.infrastructure.cluster.x-k8s.io)] [VCFMS-HEALTH-002], referenceCode=#######}}, stageId=########-####-####-####-###########}


Environment

• VCF 9.1
• VIDB 9.1

Cause

• This issue occurs when VIDB nodes are not in a Ready state and the Package Deployment is In progress or failed.
• If there are any configuration changes on the VIDB Cluster configuration, the VIDB Cluster VMs perform a rollout to apply the new configuration. This process is expected to take around 30 minutes and the Nodes will not be in a Healthy state during this state.
  
  For example:
  
  Following the KB Error: "Source and target systems use different datastores" - Upgrade of VCF Components from 9.0 to 9.1 fails during Import Validation Precheck results in the cluster nodes rollout operation and immediately retrying the upgrade results in this issue.

Resolution

Follow the steps below to check the PackageDeployment status in VIDB configuration and retry the upgrade:

1. Log in to the VIDB appliance using SSH with the account vmware-system-user.
2. Once logged in, use the su or sudo command to elevate to root and enter the password when prompted:
   
   sudo su -
   
   
3. Run the following kubectl command to check Package Deployment Status:
   
   kubectl get pd -A
   
   Sample result:
   
   root@<hostname> [ ~ ]# kubectl get pd -A
NAMESPACE         NAME                 AGE   PHASE        STATUS
.
.
vmsp-platform     vmsp-platform        ##d   Successful   successful package deployment
   
   
4. Check the status of "vmsp-platform" PackageDeployment, it should be in Successful state to proceed with the upgrade.
   1. If it is Progressing state:
      1. Wait for the rollout operation to complete (status: successful)
      2. Retry the upgrade
   2. If it Failed state:
      1. If the PackageDeployment is in Failed state, it requires further investigation to identify the reason for the failure.
      2. Contact Global Support to investigate the reason for the failure.

Additional Information

• Error: "Source and target systems use different datastores" - Upgrade of VCF Components from 9.0 to 9.1 fails during Import Validation Precheck.
• VIDB Upgrade to 9.1 fails at Power Off Source Cluster step with Error vmsp_upgrade_vidb failed.