GSLB service impact during upgrade if only Control plane HM is enabled for the GS

Products

• VMware Avi Load Balancer

Issue/Introduction

• DNS resolution respond with an IP of lower priority site for a few seconds during the upgrade of GSLB leader site and GSLB is placed in maintenance mode.
• This is the case if no data path health monitor is configured on affected GS services.

Environment

Avi version <= 30.2.2

Cause

• Error message noticed in glb remote worker logs 

  [glb_mgr_events.glb_mgr_exception_handler:171] ^[[31mURL Post Error:gslbsiteops/siteconfig/gslb-follower-xxxx:4:('HTTP Error: 500 Error Msg <h1>Server Error (500)</h1>', <Response [500]>):AviServerError:Traceback (most recent call last):   File "/opt/avi/python/bin/glb_mgr/glb_mgr_site_ops.py", line 225, in _send_msg     status, msg = self._parse_response(mtype, msg, rsp.json())   File "/opt/avi/python/lib/avi/sdk/avi_api.py", line 122, in json     raise AviServerError('HTTP Error: %d Error Msg %s' % ( avi.sdk.avi_api.AviServerError: ('HTTP Error: 500 Error Msg <h1>Server Error (500)</h1>', <Response [500]>)

• As the control plane status gets mapped to OPER_UNKNOWN during this time period, so we send this status to SE and requests are getting pushed to low priority pool member. 
• This is an expected behavior if GS is configured only with control plane HM. 

Resolution

• Configure Datapath HM: A Datapath health monitor (HM) in Avi GSLB actively probes the health of GSLB pool members (virtual services) from the Service Engine (SE) running the DNS virtual service. This ensures that only healthy endpoints are returned in DNS responses.
• To configure a datapath health monitor for a GSLB service in Avi, create a health monitor profile, attach it to the GSLB service, ensure network reachability, and verify the health status. This enables active, protocol-based health checks from the DNS SE to all GSLB pool members.