VIDB Upgrade to 9.1 fails at Power Off Source Cluster step with Error vmsp_upgrade_vidb failed

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• VIDB to 9.1 fails during Perform Component Upgrade stage after completing the upgrade pre-checks.
• You see the following error messages during the failure:
  
  Power Off Source Cluster. Status: Failed

Step vmsp_upgrade_vidb failed.


  Unexpected error occurred while checking task: Error while extracting response for type [com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]

  An unexpected error occurred in step vmsp_upgrade_vidb. Reference Code: ########. Please contact support with this reference code. Detail: Unexpected error occurred while checking VMSP task: Error while extracting response for type [com.broadcom.vcf.lcm.sddc.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]

  
  
  

Environment

• VIDB 9.1
• VCF 9.1

Cause

This issue occurs due to a path discrepancy between the vCenter inventory and the VIDB Cluster configuration. Specifically, triggered when you manually relocate VIDB 9.0 Cluster VMs to a different VM folder within vCenter post-deployment, invalidating the original path stored in the cluster's configuration.

Resolution

Use the following steps to check the VIDB cluster configuration and relocate the VIDB 9.0 VMs to the correct VM folder path:

1. Log in to the VIDB appliance using SSH with the account vmware-system-user.
2. Once logged in, use the su or sudo command to elevate to root and enter the password when prompted:
   
   sudo su -
   
   
3. Run the following kubectl command to check the Folder path in VIDB cluster configuration:
   
   kubectl get pd vmsp-platform -n vmsp-platform -o jsonpath='{.spec.values.provider.vsphere}' | jq | grep  folder
   
   Note: The VM folder path will be in the format <Datacenter Name>/vm/<Parent Folder Name>/<Child Folder Name 1>/<Child Folder Name 2>..
   
   Sample Result:
   
   kubectl get pd vmsp-platform -n vmsp-platform -o jsonpath='{.spec.values.provider.vsphere}' | jq | grep  folder

  "folder": "/datacenter01/vm/Discovered Virtual Machines",
   
   
4. Record the VIDB VM folder path from above step.
5. Login to vCenter Server using vSphere Client and select the VMs and Templates view.
6. Search for the VIDB Cluster VMs.
7. Check the folder path of these VMs.
   1. In the sample screenshot below, the folder path is datacenter01/vm/vidb-vms
      
      
      
      
8. If the folder path is different, relocate the VIDB cluster VMs (drag and drop) the VMs to the folder path configured in VIDB cluster configuration, which is recorded in Step 5 above.
   
   Note: In case the folder path is not available on vCenter, create the folders with same name as the folder stored in VIDB cluster configuration and then, relocate the VMs.
   
   
9. Retry the upgrade after relocating the VMs to the correct folder path.