vCenter update task fails on VCF 4.5 or 5.0 with error: Could not convert field “task” of structure ‘com.vmware.appliance.update.into’

Products

• VMware Cloud Foundation
• VMware vCenter Server

Issue/Introduction

Symptoms:
When attempting to update the vCenter Server, the following errors are seen in lcm.log

SDDC Manager - /var/log/vmware/vcf/lcm/lcm.log
 

‘Could not convert field “task” of structure ‘com.vmware.appliance.update.info’
2021-09-15T20:17:48.610+0000 ERROR [vcf_lcm,739578f921971a20,30db,upgradeId=6d513976-36e0-43e3-8afc-e5409f89e767,resourceType=VCENTER,resourceId=06b73bd8-7f1a-4cc6-97a5-c72a16d4ed29,bundleElementId=b57dd127-c977-4656-bff6-24c6fb069a8f] [c.v.e.s.l.p.i.vcenter.VCenterUpdater,ThreadPoolTaskExecutor-10] VCenter upgrade stage failed with exception
com.vmware.vapi.client.exception.BindingsException: Could not convert field 'task' of structure 'com.vmware.appliance.update.info'

exception com.vmware.vapi.client.exception.BindingsException: Could not convert field 'task' of structure 'com.vmware.ap'

Structure com.vmware.appliance.task_info has a union constraint with tag 'status' of value 'PENDING', for which the field 'progress' is not allowed to be set.

Environment

Vmware Cloud Foundation 4.5
VMware vCenter Server 7.0.3

Cause

The issue results from a problem with the vCenter Server patching API, which returns a response that violates the contract, and in turn, the client-side bindings complain about it.

The Info in the response of this operation has a task field of TaskInfo type, which in turn includes the CommonInfo.

The error message states the union constraint in that response is violated since the tag status=PENDING, but the field 'progress' is present, while it is not allowed to be for this tag value per the definition.

Resolution

This issue is resolved when the source vCenter version is 8.0.2.0, also available as a part of VCF 5.1 BoM.

To recover from this issue, vCenter needs to be recovered from the backup. Please refer to Restore vCenter Server from a File-Based Backup for more information on how to restore vCenter from backup.

Workaround:

As a workaround, follow the below steps.

1. Check if vCenter is already upgraded to the target by logging into the vCenter VAMI page:

https://<VC_FQDN>:5480

2. Retry the upgrade from SDDC Manager. The upgrade should skip the stages after pre-validation until installed, resume from post-validation, and complete successfully.

If we continue to run into the same error, proceed with the following steps:

3. Download the relevant vCenter Patch ISO from customer connect: https://my.vmware.com/group/vmware/patch

Note: Alternatively, you can acquire the ISO from the NFS Mount on the SDDC Manager.

4. Upload the ISO to a non-vSAN datastore that the ESXi host (where the vCenter VM resides) can access.

5. Login to the vCenter vSphere UI > Navigate to the vCenter VM > Edit Settings > CD/DVD Drive > Datastore ISO > Click on the datastore and select the ISO > Then hit the connect checkbox.   

6. Once the ISO has been mounted onto the vCenter VM, SSH into the vCenter with the root user.

7. In the appliance shell (not Bash Shell) run the following commands:

 software-packages unstage
software-packages stage --iso --acceptEulas
software-packages install --staged --acceptEulas

Note: Do not reboot the vCenter after the update completes.

8. Once the patch completes successfully, go back into the SDDC manager and re-try the upgrade, which should complete successfully.