Log Insight load balancer incompatible with NSX Distributed Firewall Protection

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• In the vRealize Log Insight user interface under Administration > Management > System Monitor > Statistics, you see these symptoms:
   
  • The Syslog Events Incoming Rate (Per Second) metric is higher on the Integrated Load Balancer (ILB) node relative to all other nodes.
  • The API Events Incoming Rate (Per Second) metric is higher on the ILB node relative to all other nodes.
     
• Higher CPU usage on the ILB node as compared to other nodes in a vRealize Log Insight cluster.
• You are using vRealize Log Insight and VMware NSX.
• A Log Insight Cluster is behind NSX distributed Firewall Protection.

Environment

VMware vRealize Log Insight 4.x
VMware vRealize Log Insight 3.x
VMware vRealize Log Insight 8.x

Cause

This issue occurs because vRealize Log Insight's cluster virtual IP uses a Linux Virtual Server in Direct Server Return Mode (LVS-DR) for load balancing which is not supported by NSX.

Resolution

This is an expected behaviour for vRealize Log Insight in use with NSX.

 

To work around this issue, exclude the virtual machines that are part of the vRealize Log Insight cluster from VMware NSX Distributed Firewall Protection by following the below steps.

For NSX-T: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-3B3C278D-4E35-4CE9-A4E2-ED6B1F25ABCE.html
For NSX-V: https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.install.doc/GUID-C3DDFBCE-A51A-40B2-BFE1-E549F2B770F7.htm

After following the steps relevant to your version of NSX above, it is necessary to perform a reboot of all nodes in the vRealize Log Insight cluster.

​​​​​​Note: A service restart will not suffice here, a reboot is required.

1. In the vRealize Log Insight UI, note which node has the ILB (Integrated Load Balancer). You will reboot this node last.
2. From the vSphere Web Client, right click the vRealize Log Insight node, select Power > Restart Guest OS
3. Repeat step 2 on the remaining nodes in the cluster, one by one.