[Internal] "Failed to sync to NTP servers" "com.vmware.applmgmt.err_ntp_sync_failed' while upgrading VCSA, it gets stucks at 50%

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VCSA upgrade stuck at 50% without any failure.
• In /var/log/vmware/bootstrap.log, you see entries similar to,
• In upgrade.log, you see entries similar to,

stderr:
2019-05-28T18:24:10.958Z ERROR apply_networking No active IP addresses found on interface eth0:0

2019-05-28T18:24:11.973Z ERROR transfer_network Fatal user displayable error during upgrade network transferred.
Traceback (most recent call last):
 File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/transfer_network.py", line 1218, in execute
   revertNetworkingScript)
 File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/transfer_network.py", line 787, in _applyNetworking
   applyLegacyNetworking(networkConfig)
 File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py", line 763, in applyLegacyNetworking
   _raiseSetnetError(error)
 File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py", line 673, in _raiseSetnetError
   problemId=problemId)
apply_networking.UserUpgradeNetworkConfigError: [Failed to set the time via NTP. Details: Failed to sync to NTP servers.. Code: com.vmware.applmgmt.err_ntp_sync_failed, Could not set up time synchronization.]

2019-05-28T18:24:11.978Z ERROR __main__ ERROR: Fatal error during upgrade EXPORT. For more details take a look at: /var/log/vmware/upgrade/export-upgrade-runner.log

Environment

VMware vCenter Server Appliance 8.0.x

Cause

Source vCenter Server time is not in sync with ESXi host or PSC & VCSA time is not in sync.

or 

Route not set on destination vCenter.

Resolution

Workaround 1: Remove the NTP Configuration on PSC and VCSA and try to sync with ESXi host itself. Or make sure the NTP server and PSC/VCSA are in sync.

1. To isolate if the NTP is having issue, try to run the command to reconfigure the NTP Server using below command,
   • # com.vmware.appliance.ntp.set --servers <NTP Server IP Address>
   •   Error in method: Failed to sync to NTP servers. (code  com.vmware.applmgmt.err_ntp_sync_failed)
2. If there is any issue you might receive similar error as mentioned above.
3. Try to remove the NTP settings temporarily and then sync the PSC/VCSA node with ESXi host and then retry the install/upgrade. 
   • To check if the NTP settings is already there, run the below command from bash shell
     • com.vmware.appliance.ntp.get
   • To remove the NTP settings, remove the last line in the below file /etc/ntp.conf file

root@vcsa [ ~ ]# cat /etc/ntp.conf
tinker panic 0
restrict default kod nomodify notrap nopeer
restrict 127.0.0.1
restrict -6 ::1
driftfile /var/lib/ntp/drift/ntp.drift
servers: 10.10.11.22 ---> Remove this line

Workaround 2: When the upgrade installer reaches 50% you need to manually check the destination VCSA DCUI console and make sure its having the correct IP address, if not manually enter IP, subnet mask and default gateway. 

1. Go to command prompt from the machine where you are trying to upgrade the VCSA to check if the temporary IP is pingable
   • ping -t <temporary_ip>
2. Manually open a vm console to the deployed target/destination appliance the moment the installer reaches 50%
3. Press F2 to login to the DCUI console using root password
4. Go to configure management network > IP configuration 
5. Under static IP configuration validate if the correct IP address, subnet mask and default gateway details.
6. If the IP's differ from those entered in stage 1, manually change the values and save the configuration and retry the installation/upgrade.