Avi Backup Config Files Missing in Avi Load balancer UI

Products

• VMware Avi Load Balancer

Issue/Introduction

• Avi Load Balancer config backup not show in UI
• Avi Load Balancer config backup out of sync with follower node
  
  

Cause

• The configuration backup created on the Leader node doesn't sync to the Follower node or appear in the Controller UI if the system time is not in sync with the NTP server. 
• This issue could happen if the Controller node can't reach the NTP server or if the network firewall blocks UDP port 123.
• As a result, the backup configuration won't sync across all nodes or display in the UI as expected.
  
  
• Example:
  
  
  
  
• In-Spite of the avi config backup present on the controller leader it is not visible in the UI
  

  root@xx-xx-xx-xx:/var/lib/avi/backups# ls -l
  total 18816
  -rw-r--r-- 1 root root 4865034 Month DD 15:52 backup_Default-Scheduler_YYYYMMDD_HHMMSS.json
  -rw-r--r-- 1 root root 4852392 Month DD 15:52 backup_Default-Scheduler_YYYYMMDD_HHMMSS.json
  -rw-r--r-- 1 root root 4776040 Month DD 15:52 backup_Default-Scheduler_YYYYMMDD_HHMMSS.json
  -rw-r--r-- 1 root root 4763800 Month DD 15:52 backup_Default-Scheduler_YYYYMMDD_HHMMSS.json

Resolution

• How to identify the NTP Issue
  To identify the NTP out of sync, when you login to the controller from the browser and just after you authenticate, you will see a pop-up box on the top which will tell you if the NTP time sync is not correct.

• Apart from that, check the current time with NTP (Network Time Protocol) use below API
  https://<controller-ip>/api/initial-data
• From the API output, check the  controller node time "current_time" field in UTC. Please convert into local timezone and compare with your desktop/system time and check the difference
  
  
  
• You can also check from controller bash, login into Controller leader node and follower node and run the below commands to make sure that time is sync or not and NTP service is running
  

  # timedatectl
  # systemctl status ntp.service

• After that you can run below command to check if controller able to sync the time with configured NTP servers
  

  root@:/home/admin# ntpq -pn
  
       remote           refid      st t when poll reach   delay   offset  jitter
  ==============================================================================
  xx.xx.xx.xx     .INIT.          16 u    - 1024    0    0.000    0.000   0.000
  yy.yy.yy.yy     .INIT.          16 u    - 1024    0    0.000    0.000   0.000
  

• Here stratum would be 16 only when controller is not able to reach the configured NTP servers or the NTP server it is trying to reach is having a stratum of 16.
• Please check network connectivity in your infra and make sure that UDP Port 123 allowed for all 3 controller IP address. Here's few command to check NTP server connectivity.
  

  >> Ping Test from Controller to NTP Server IP
  root@:/home/admin#  ping ntp.ubuntu.com
  
  >> Check UDP Port 123 allow from Controller to NTP Server
  echo "Test message" | nc -u ntp.ubuntu.com 123
  
  If the UDP port is reachable and open, the message sent from the client will appear on the server's terminal.
  
  If there’s no response, it might indicate that the port is blocked by a firewall, or the service isn't reachable. Note that UDP is connectionless, so you won't receive an acknowledgment like with TCP.