Upgrading VCF from 4.1 to 4.2 fails at pre-checks with NTP TIME SYNC CHECK
Article ID: 344826
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
Error Description:
NTP time sync check failed as VC time is not in sync with SDDC VM
Remediation:
Check if VC is down.
Check if VC/SDDCMANAGER time is in sync with NTP server.
If logged into PSC/VC on the Appliance shell, switch to bash shell using command 'shell' and try running the command 'date +%s' on target VM BASH shell. Make sure that the SDDCMANAGER and VC time are in sync.
Troubleshooting:
-NTP Health check was GREEN
-Checked the data "date %s" on both VC and SDDC Manager.
-Tried restarting ntp service on both SDDC-Mgr and VC side. Failure still persisted.
Error Description:
NTP time sync check failed as VC time is not in sync with SDDC VM
Remediation:
Check if VC is down.
Check if VC/SDDCMANAGER time is in sync with NTP server.
If logged into PSC/VC on the Appliance shell, switch to bash shell using command 'shell' and try running the command 'date +%s' on target VM BASH shell. Make sure that the SDDCMANAGER and VC time are in sync.
Troubleshooting:
-NTP Health check was GREEN
-Checked the data "date %s" on both VC and SDDC Manager.
-Tried restarting ntp service on both SDDC-Mgr and VC side. Failure still persisted.
Environment
VMware Cloud Foundation 4.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.0.x
Cause
Log snippet:
lcm-debug.log:
2021-04-13T18:41:08.761+0800 DEBUG [vcf_lcm,0000000000000000,0000,precheckId=dc0d2e32-a0f1-4fd6-92a1-c2b57327bf67,resourceType=COMMON_SERVICES,resourceId=a61
54964-2672-4932-bfec-7f1a3077823f] [c.v.e.s.l.p.e.util.spi.PrimitiveUtil,pool-3-thread-1] epochTimeOnVCenter(cfbcff7a-0156-4eb9-8caa-3aaae06a2bd4): 1618310463000, timeInSddcVM: 1618310410
NTP_TIME_SYNC_CHECK is failing as the difference of time on vcenter and sddcVM is more than 45 sec
The VC/PSC upgrade fails sometimes if the time difference is more than the required time and hence LCM added the check.
LCM gets all the Vcenters from the inventory and checks NTP time for each of them with SDDC VM time. If any of the Vcenter time is more than 45 secs than SDDC time, the precheck task NTP_TIME_SYNC_CHECK will fail which could be because of the network latency.
lcm-debug.log:
2021-04-13T18:41:08.761+0800 DEBUG [vcf_lcm,0000000000000000,0000,precheckId=dc0d2e32-a0f1-4fd6-92a1-c2b57327bf67,resourceType=COMMON_SERVICES,resourceId=a61
54964-2672-4932-bfec-7f1a3077823f] [c.v.e.s.l.p.e.util.spi.PrimitiveUtil,pool-3-thread-1] epochTimeOnVCenter(cfbcff7a-0156-4eb9-8caa-3aaae06a2bd4): 1618310463000, timeInSddcVM: 1618310410
NTP_TIME_SYNC_CHECK is failing as the difference of time on vcenter and sddcVM is more than 45 sec
The VC/PSC upgrade fails sometimes if the time difference is more than the required time and hence LCM added the check.
LCM gets all the Vcenters from the inventory and checks NTP time for each of them with SDDC VM time. If any of the Vcenter time is more than 45 secs than SDDC time, the precheck task NTP_TIME_SYNC_CHECK will fail which could be because of the network latency.
Resolution
Workaround:
- Note: Please take a snapshot of the SDDC VM prior performing the below change.
- Change the property value (lcm.ntp.acceptable.timedifference=150) in lcm/lcm-app/conf/application-prod.properties
Add the following entries in /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
##### LCM NTP Properties #####
lcm.ntp.acceptable.timedifference=150
##### LCM NTP Properties #####
lcm.ntp.acceptable.timedifference=150
- Restart lcm with the command "systemctl restart lcm"
Feedback
Yes
No
Powered by
