Pools marked down after enabling health monitor for virtual service when the ICMP is blocked in the infra in VMware AVI Load Balancer

Products

• VMware Avi Load Balancer

Issue/Introduction

After configuring or enabling a health monitor for a virtual service, backend pools are unexpectedly marked down when ICMP is blocked in the environment for the default gateway.

Environment

VMware AVI Load Balancer

Cause

By default, gateway monitor is enabled, which uses ICMP (ping) requests directed at the default gateway. Upon receiving a successful response, the system marks the default gateway routes as up (GWUp).

If ICMP traffic is blocked by network firewalls or security policies in the environment:

1. The internal health monitor fails to ping the default gateway.

2. The default route is not marked as active/up.

3. As a result, no health monitor will be initiated towards the pool members, hence marking the pool down.

 

Resolution

There are 2 ways to fix the issue:

1. Either, by enabling the ICMP in the environment for the default gateway,
2. Or, disabling the internal gateway monitor on Avi.

Steps to disable the internal gateway monitor.

1. Step 1: Disable the Gateway Monitor via CLI
   1. SSH into the controller CLI and input the following commands to modify your specific VRF context:
   2. Shell:

      [admin:##.##.##.##]: > configure vrfcontext <vrfcontext_name>
      [admin:##.##.##.##]: vrfcontext> internal_gateway_monitor 
      [admin:##.##.##.##]: vrfcontext:internal_gateway_monitor> disable_gateway_monitor 
      [admin:##.##.##.##]: vrfcontext:internal_gateway_monitor> save
      [admin:##.##.##.##]: vrfcontext> save
      

2. Step 2: Reboot the Service Engine (SE)
   1. The configuration change requires a restart of the service engine

      1. Schedule a maintenance window.

      2. Reboot the affected Service Engine (SE) hosting the virtual service.

      3. Once the SE reboots, the default gateway route will accurately transition to an up (GWUp) status without relying on ICMP traffic, restoring the pool status to green.

3. Validate:
   1. Log in to the controller shell
      1. show serviceengine <se_name1> route filter vrf_ref <vrf_name>
   2. Before disabling the gateway monitoring
      
      
      After disabling the gateway monitoring and after rebooting the SE.