Cisco ACI based environment conflicting with the VIP in Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• The Administration > Management > System Monitor page intermittently shows Failed to load resources and agents are reporting Disconnected in a Cisco ACI based environment.
• Aria Operations for Logs VIP is not reachable for ingestion, however ingestion sent directly to a node succeeds.
• The /var/log/loginsight-agent/liagent_latest-date.log contains the following error
  • Transport error while trying to ingest SSL connect error
• The application web page loads only sometimes and other times it does not load when accessing it using the VIP. However, you can access vRLI using the IP of the nodes

Environment

Aria Operations for logs 8.x
VMware vRealize Log Insight 4.x
VMware vRealize Log Insight 8.x

Cause

• The Aria Operations for Logs load balancer uses a Direct Server Return (DSR) configuration.
• By default, DSR does not work in Cisco ACI because of data-plane IP learning.

Resolution

The L4-L7 Virtual IPs option was introduced in Cisco Application Policy Infrastructure Controller (APIC) Release 1.2(1m).
This option is located at Tenant > Application Profiles > Application EPGs or uSeg EPGs.
This option disables data-plane IP learning for the specific DSR virtual IP address. Failure to disable IP learning for the DSR virtual IP address will result in IP endpoint flapping between different locations in the Cisco ACI fabric.

For more information, see ACI Fabric Endpoint Learning White Paper.

 Note:

• Ensure that GARP is set to "enabled" for the segment on CISCO ACI. The default GARP setting is "disabled."
• Some versions of Cisco ACI appliances use an option labelled 'IP Data-plane Learning', set this to 'no' as there is no GARP option.
  •  
  • 

 

 

After following the above steps, it is necessary to perform a reboot of all nodes in the Aria Operations for Logs cluster.

​​​​​​Note: A service restart will not suffice here, a reboot is required.

1. In the Aria Operations for Logs UI, note which node has the ILB (Integrated Load Balancer). You will reboot this node last.
2. From the vSphere Web Client, right click the Aria Operations for Logs node, select Power > Restart Guest OS
3. Repeat step 2 on the remaining nodes in the cluster, one by one, waiting for each node to come online before moving to the next node.

Additional Information

In the event that this Resolution is not working for Aria Operations for Logs in Cisco based environments, the issue could be due to a rather complex ACI design with intra-EPG isolation enabled and micro-segmented (uSeg) EPGs.

With IP data-plane learning disabled, the fabric depends on ARP/GARP and COOP to move the VIP. In an intra-EPG isolated and micro-segmented design, those ARP/GARP signals can be proxied, suppressed, or scoped by policy (e.g., per-EPG contracts, ARP suppression on the Bridge Domain). If the new ILB owner's GARP doesn't qualify as a valid "move" in the same policy context, the leaf keeps the stale EPM binding, so the VIP appears stuck on the old node.

Also, because uSeg EPGs split endpoints into multiple policy scopes, the "old owner" and "new owner" can be in different uSeg EPGs. ACI may hold separate endpoint state per scope; without the right GARP handling and Bridge Domain settings, the old record doesn't get replaced, even though traffic is flowing.

In such Cisco network configurations, the standard load balancer / VIP functionality in Aria Operations for Logs will not work. The only workaround is to use an external load balancer such as F5 or DNS-based balancing.

NOTE: The configuration of external load balancing options is out of the scope of Broadcom support.

Please see Microsegmentation with Cisco ACI for more details and limitations.

"Configuring a Layer 4 to Layer 7 virtual IP (VIP) address under microsegmented EPGs or their corresponding base EPGs is not supported."