Aria Operations local admin account is locking out when behind Native NSX load balancer

Products

• VMware NSX

Issue/Introduction

• Aria Operations cluster is sitting behind an NSX Native Load Balancer.
• The Load Balancer is configured in One Arm mode.
• The KB Aria Operations admin account getting locked even after password reset has been reviewed and identified the IP addresses.
• In the Aria Operations logs, the API calls are coming from the IP address of the NSX Load Balancer service interface, as can be seen in log /storage/log/vcops/log/http_api.log:

INFO [ajp-nio-127.0.0.1-8010-exec-3, ###################] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=<Load Balancer IP>, payload={"username":"admin","password":"######"}]

Environment

Aria Operations

VMware NSX

Cause

One of the clients making calls to Aria Operations, via the NSX Native Load Balancer, has the incorrect admin password and leads to the account being locked in Aria Operations.

As this is a one arm Load Balancer, by default, the IP address on the service interface of the Load Balancer, is used for the backend connection and is seen as making the API calls in the Aria Operations logs.

Resolution

Review all clients making API calls to the Aria Operations server(s) via the NSX Load Balancer and ensure they are using the correct password.

To help identify the clients making the API calls to the Aria Operations servers, via the NSX Load Balancer, you can do packet captures on the edge node where the Load Balancer is Active.

1. To identify the Active edge node, in NSX go to Networking > Load Balancing, Identify the Load Balancer which has the Virtual Server used by Aria Operations.
2. This Load Balancer will have a Attachment connected to a Tier 1 logical router, note this Tier 1 logical router.
3. Then go to Networking > Tier-1 Gateways and click on the HA Mode for the Tier 1 logical router.
   • This will show the edge nodes the logical router is deployed on, in Active/Standby mode, note the Active edge node.
4. In vCenter identify the ESXi where the Active edge node resides and then ssh as root user to that host.
5. On the ESXi host, run: net-stats -l
   • Identify the switchports, under column PortNum used for the edge node in the list, eth0 is management and can be ignored.
6. These ports will be both uplink (from the client to the server) and downlink (from the Load Balancer to the server).
7. On each of the ports run:
   pktcap-uw --switchport <switchport> --dir 2 --ip <Load Balancer virtual server IP address> -o /<path-to-save-file>/switchport-#.pcap
8. For example it may look like:
   pktcap-uw --switchport 1234567 --dir 2 --ip 192.168.1.1 -o /data/switchport-1.pcap
9. Then review the packet capture to identify the client(s) making the calls to the virtual server:
   tcpdump-uw -r /<path-to-save-file>/switchport-#.pcap
10. For example:
    tcpdump-uw -r /data/switchport-1.pcap
11. And look for the source IP addresses.
12. Or export the packet capture and open in Wireshark and filter on ip.src == <Load Balancer virtual server IP address>
13. In Wireshark there is a option in Statistics, Conversations, you can use this to quickly list all the flows to the virtual server, allowing you to identify the source IP to review.