VCF vCenter RDU upgrade failed at the VCENTER_UPGRADE_SWITCHOVER stage

Products

• VMware SDDC Manager

Issue/Introduction

An RDU upgrade of a vCenter Server through VCF / SDDC fails during the switch-over phase

From the lcm-debug.log file on the SDDC Manager you may see an error similar to the following:
Upgrade error occured: vCenter upgrade failed at the VCENTER_UPGRADE_SWITCHOVER stage  /  Unable to start all services during the upgrade

From the vlcm.log on the source vCenter Server you may see a similar error: 
Error type: map[Data:null] ErrorType:"ERROR" Messages:[{"Id":"vcenter.deployment.migration_upgrade.unable_to_start_services_error","DefaultMessage":"Unable to start all services during the upgrade","Args":[],"Params":null,"Localized":"Unable to start all services during the upgrade"},

Further review of the RDU logs shows that VPXD failed to start:
error vlcm [logger/multiline.go:45] [ObjectID=108] OUT:001 Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

Reviewing the RDU vpxd.log shows that we have no space for the VCDB:
error vpxd[128495] [Originator@6876 sub=vpxdVdb] Shutting down the VC as there is not enough free space for the Database(used: 96%; threshold: 95%).

 

Environment

VCF 9.0

vCenter Server 9.0

 

Cause

The RDU automatic sizing functionality fails to take into account manually expanded VMDKs for the target vCenter Server appliance.  

Resolution

This issue will be resolved in a future release.  To work-around this issue, truncate the task / event / performance data from the VCDB to bring the size back down below threshold. See KB313506 for details on truncating the VCDB tables. 

Additional Information

Delete old tasks, events and statistics data in vCenter Server 5.x, 6.x, 7.x and 8.x