VCF Automation Upgrade to 9.1 fails at Power Off Source Cluster step with Error vmsp_upgrade_vcfa failed

Products

• VCF Automation

Issue/Introduction

• VCF Automation upgrade to 9.1 fails during Perform Component Upgrade stage after completing the upgrade pre-checks.
• You see the following error messages during the failure:
  
  Power Off Source Cluster. Status: Failed

Step vmsp_upgrade_vcfa failed.
  
  

Environment

• VCF Automation 9.1
• VCF 9.1

Cause

This issue occurs due to a path discrepancy between the vCenter inventory and the VCFA Cluster configuration. Specifically, triggered when you manually relocate VCFA 9.0 Cluster VMs to a different VM folder within vCenter post-deployment, invalidating the original path stored in the cluster's configuration.

Resolution

Use the following steps to check the VCFA cluster configuration and relocate the VCFA 9.0 VMs to the correct VM folder path:

1. Log in to the VCFA appliance using SSH with the account vmware-system-user.
2. Once logged in, use the su or sudo command to elevate to root:
   
   sudo su -
   
   
3. Enter the root password when prompted.
4. Run the following kubectl command to check the Folder path in VCFA cluster configuration:
   
   k get pd vmsp-platform -n vmsp-platform -o jsonpath='{.spec.values.provider.vsphere}' | jq | grep  folder

Note: The VM folder path will be in the format <Datacenter Name>/vm/<Parent Folder Name>/<Child Folder Name 1>/<Child Folder Name 2>..

Sample Result:

k get pd vmsp-platform -n vmsp-platform -o jsonpath='{.spec.values.provider.vsphere}' | jq | grep  folder

  "folder": "/datacenter01/vm/Discovered Virtual Machines",
   
   
5. Record the VCFA VMs folder path from above step.
6. Login to vCenter Server using vSphere Client.
7. Select the VMs and Templates view.
8. Search for the VCFA Cluster VMs.
9. Record the VCFA VM folder path.
   1. In the sample screenshot below, the folder path is datacenter01/vm/vcf-automation
      
      
      
      
10. If the folder path is different, relocate the VCFA cluster VMs (drag and drop) the VMs to the folder path configured in VCFA cluster configuration, which is recorded in Step 5 above.
    
    Note: In case the folder path is not available on vCenter, create the folders with same name as the folder stored in VCFA cluster configuration and then, relocate the VMs.
    
    
11. Retry the upgrade after relocating the VMs to the correct folder path.