In some cases, the AppLogic BFC, controller, appliance and physical node(dom0) may have incorrect time, this document introduces basic troubleshooting procedure to address such sort of problem.
In AppLogic 2.x and 3.0, controller is solo ntp server in the grid, time sync flow is like below. Ntp on controller and physical node processes the time sync up, and hypervisor is in charge of passing time from physical node to appliance VM.
controller<= physical node <= appliance VM
From 3.1, BFC take the role of controller and become the ntp server of all grids managed by it, time sync flow is changed to the following procedure. Ntp on BFC and physical node processes the time sync up, and hypervisor is in charge of passing time from physical node to controller and appliance VM.'
BFC<=physical node<= controller and application VM
If external ntp server is configured from BFC GUI, time sync flow should look like the below
external ntp server<=BFC<=physical node<= controller and application VM
Another major change from 3.1 is all physical node clock (both system clock and hwclock) use UTC+0 time as opposed to local time. Time drift sync with BFC to physical node is also based on UTC time.
If you would like know more details of ntp, please refer to following link
The time synchronization issue of any link of the chain may result in bfc, controller, appliance VM or physical node of next links have incorrect time.
Please follow the below check list to locate which part has time time synchronization issue
Check list for AppLogic 2.x and 3.0
Check list for AppLogic 3.1 and newer release
Note: there is known time sync issue in 3.1 due to a Xen time drift bug in which physical node(dum0) has trouble to pass time drift to hypervisor, the end result is controller and appliance VM has incorrect time. The solution is set independent wall clock in appliance VM, additionally, install and configure ntp to sync time from either BFC or external ntp server.
How to identify the time sync with external ntp works properly
This section applies to controller in AppLogic 3.0 and prior release , as well as BFC of 3.1 and newer release if external ntp server is configured.
Note: When configuring ntp server in the BFC GUI 3.1 and newer release, you may input any valid and available external ntp server, but not BFC name or ip address.
remote refid st t when poll reach delay offset jitter ============================================================================== *ntpsrv1 184.108.40.206 4 u 995 1024 377 0.411 0.019 0.031 ntpsrv2 220.127.116.11 5 u 708 1024 377 46.798 -0.100 0.051
How to identify the time sync to physical node work properly
If physical node time are different to the solo ntp server in the grid(controller in 3.0 and prior release, BFC in 3.1 and newer release), similarly, we can also utilize the following approaches for verification and troubleshooting.
Note: From 3.1, both system time and hardware clock time of physical node should be UTC+0 time, and they should not have significant gap. BFC is still local time. For instance, current time on BFC is 20:00 PM (UTC+10 time zone), node time is 10:00 AM (UTC+0), in such case, their time are consistent.
[[email protected] srv1 ~]# date
Thu Jun 28 06:19:37 UTC 2012 -> OS system time
[[email protected] srv1 ~]# hwclock
Thu 28 Jun 2012 06:19:38 AM UTC -0.549140 seconds -> hardware clock
The time zone of physical node system is stored in /etc/localtime, it should either set as UTC like below or link to a /usr/share/zoneinfo/UTC
[[email protected] srv1 ~]# cat /etc/localtime
The time zone of hardware clock is stored in /etc/sysconfig/clock as below.
[[email protected] srv1 ~]# cat /etc/sysconfig/clock
How to identify the time of appliance VM correct or not
Basically, if physical node time is correct, the appliance VM time should be correct as long as its time zone is configured as correct local time zone. If appliance VM time is incorrect, following information may help to address the problem