vCenter Upgrade Fails in Stage 2 due to inability to configure NTP on target appliance

Products

VMware vCenter Server

Issue/Introduction

vCenter upgrade from 7.x to 8.x fails at Stage 2 with errors indicating time synchronization could not be set up with NTP:

Could not set up time synchronization
Verify that provided ntp servers are valid.

Failed to set the time via NTP. Details: Failed to sync to NTP servers. Code:
com.vmware.applmgmt.err_ntp_sync_failed.
Verify that provided ntp servers are valid

The /var/log/vmware/upgrade/bootstrap.log contains following log snippets:

[YYYY-MM-DDTHH:MM:SS].845Z INFO transfer_network Preserving time sync configuration {'VC_TIMESYNC_NTP_SERVERS': '<NTP_Server_IP-address>', 'VC_TIMESYNC_TOOLS': '0', 'VC_TIMESYNC_NTP': '1'}to file /storage/seat/cis-export-folder/system-data/timesyncconfig.json
[YYYY-MM-DDTHH:MM:SS].121Z INFO install_param_utils 'appliance.ntp.servers' successfully set to <NTP_Server_IP-address>
[YYYY-MM-DDTHH:MM:SS].855Z Getting value for install-parameter: appliance.ntp.servers
[YYYY-MM-DDTHH:MM:SS].855Z setting time sync to ntp
[YYYY-MM-DDTHH:MM:SS].843Z ERROR: Failed 1/3 times. Exception: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed
[YYYY-MM-DDTHH:MM:SS].858Z setting time sync to ntp
[YYYY-MM-DDTHH:MM:SS].858Z ERROR: Failed 2/3 times. Exception: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed
[YYYY-MM-DDTHH:MM:SS].874Z setting time sync to ntp
[YYYY-MM-DDTHH:MM:SS].851Z ERROR: Failed 3/3 times. Exception: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed
"id": "install.setnet.set.timesync.ntp",
"translatable": "Failed to set the time via NTP. Details: %(0)s",
"Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed"
"localized": "Failed to set the time via NTP. Details: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed"
"id": "install.setnet.set.timesync.ntp.res",
"translatable": "Verify that provided ntp servers are valid.",
"localized": "Verify that provided ntp servers are valid."
apply_networking.UserUpgradeNetworkConfigError: [Failed to set the time via NTP. Details: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed, Could not set up time synchronization.]

Environment

vCenter 7.x
vCenter 8.x

Cause

This behavior is likely attributed to one of the following factors:

One or more configured NTP servers are not responding as expected via the NTP protocol.

A communication gap exists between the vCenter (or its temporary deployment IP) and the NTP server(s).

Resolution

Resolution:

1. Correct any connectivity issues to the NTP server(s).

2. Ensure that the NTP server is functioning properly. Confirm that vCenter is receiving time by issuing the "ntpq -p" command via an SSH session to vCenter.

3. Retry the vCenter upgrade..

Workaround:

1. Ensure the ESXi host where the source vCenter is running on has NTP enabled and is working correctly - Configuring Network Time Protocol (NTP) on an ESXi host using the vSphere Client.

2. Shutdown the failed vCenter Server instance. This instance can be deleted as after implementing the workaround as the upgrade needs to be started from started from stage-1.

3. Power-on the source vCenter Server appliance, login to the vCenter Management Interface (VAMI), select "Time" form the left menu then configure to select "Host" and not NTP or Disabled.

4. Login to the ESXi Host Client where the vCenter Server appliance is running and shutdown the vCenter Server appliance. Edit Settings of the vCenter Server appliance VM, then under VM Options --> VMware Tools, select "Synchronize Time with Host". This will ensure that the source vCenter VM is not relying on NTP to complete the upgrade.

5. Power-on the vCenter Server appliance and retry the vCenter upgrade from Stage-1.

6. Once the vCenter upgrade is complete, the newly upgraded vCenter Server appliance can be reconfigured back to use NTP. Refer: Configure Network Time Protocol (NTP) on the Vcenter