vCenter upgrade in VCF 5.1.x fails with error "SET_DRS_AUTOMATION_LEVEL_TO_MANUAL_FAILED"

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• vCenter upgrade in VCF 5.1.x fails with error

  SET_DRS_AUTOMATION_LEVEL_TO_MANUAL_FAILED

• Error in /var/log/vmware/vcf/lcm/lcm.log

  ERROR [vcf_lcm,0000000000000000,0000,upgradeId=fa13####-####-####-########bc8f9,resourceType=VCENTER,resourceId=4c90####-####-####-########6773,bundleElementId=9fef####-####-####-########b298] [c.v.e.s.l.p.i.v.VCenterUpgrader,Upgrade-1] Failed to Change DRS behavior
  java.lang.RuntimeException: VM vcsa01.example.com not found
          at com.vmware.evo.sddc.lcm.client.vmware.vsphere.VsphereUtils.findEsxHostByVmName(VsphereUtils.java:1141)
  
  
  ERROR [vcf_lcm,0000000000000000,0000,upgradeId=fa13####-####-####-########bc8f9,resourceType=VCENTER,resourceId=xxxxxxxxxx-xxxx-xxxxxxx-xxxxxx,bundleElementId=9fef####-####-####-########b298] [c.v.e.s.l.p.i.v.VCenterUpgrader,Upgrade-1] buildRemedidateSpec failed for getting the esx host vim connection
  java.lang.RuntimeException: VM vcsa01.example.com not found
          at com.vmware.evo.sddc.lcm.client.vmware.vsphere.VsphereUtils.findEsxHostByVmName(VsphereUtils.java:1141)
          at com.vmware.evo.sddc.lcm.primitive.impl.vcenter.VCenterUpgrader.buildVCenterRemedidateSpec(VCenterUpgrader.java:2365)

• Steps in KB vCenter upgrade in VCF 5.1.x fails at stage SET_DRS_AUTOMATION_LEVEL_TO_MANUAL_FAILED do not resolve the issue

Environment

5.1.X

May hit 5.2.X, highly unlikely, perhaps in scenarios where the "Synchronize inventory" task is failed.
In that case see KB Synchronize inventory versions after out-of-band upgrade in a VMware Cloud Foundation Environment

Cause

vCenter VM is vSphere UI is not matching with the name in SDDC Platform database

Resolution

1. SSH to SDDC Manager VM with vcf and su to root
2. Run the below command to find the vCenter name in SDDC Manager platform database

   /usr/pgsql/13/bin/psql -U postgres -h localhost -d platform -c "\x" -c "select * from vcenter"

   Sample output

   -[ RECORD 1 ]------------+-------------------------------------
   id                       | 4c90####-####-####-########6773
   creation_time            | 1721660071528
   modification_time        | 1738713604664
   bundle_repo_datastore    | lcm-bundle-repo
   datastore_name           | VCF-vSAN01
   ssh_host_key             |
   ssh_host_key_type        |
   status                   | ACTIVE
   type                     | MANAGEMENT
   version                  | 8.0.3.00000-24022515
   vm_hostname              | vcsa01.example.com
   vm_management_ip_address | 192.168.1.10
   vm_name                  | vcsa01
   join_sso_status          | JOINED

3. Rename the vCenter VM in vSphere UI to match with the name and vm_hostname in SDDC database.
4. Retry upgrade.

 

Additional Information

In 5.2 renaming Vcenter objects is supported - to a certain extent 
VMware Cloud Foundation 5.2 Release Notes 

Out of Band Changes from vCenter: Out of Band changes from vCenter can be manually synced with SDDC Manager.
This includes inventory changes (for example, adding a host to a cluster) and object name changes (for example, datacenter name, datastore name, port group name).

However the rename up to 5.1.x is limited see below
Renaming VCF SDDC manager objects