Customer domain controller(s) is strained due to excessive queries from VM running within vCenter/ESXi host

Products

• VMware vCenter Server

Issue/Introduction

When an application is written to send authentication requests to vCenter, the application sends the auth request to vCenter, and then vCenter forwards the request to the identity source(s) configured. When Active Directory is configured as the identity source, the requests are forwarded to the AD domain controllers. This means that the queries end up triggering communication with the DNS and other domain services.

This issue can be confirmed by checking the vpxd.log file of the vCenter Server appliance. For example, if there are is an excessive, disproportionate amount of requests for one domain user account compared to any other user account found in the /var/log/vmware/vpxd/vpxd.log file:

 

vpxd.log:

2023-12-10T11:07:16.704Z info vpxd[12345] [Originator@1234 sub=UserDirectorySso opID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Authz-9a] GetUserInfoInternal(<someDomain>\<someUserName>, false) res: <someDomain>\<someUserName>

Environment

vCenter 8.x

Cause

When an application excessively makes such authentication requests to vCenter, it can cause the domain controller(s) (and potentially the vCenter Server appliance) to be strained in trying to process the requests.

Resolution

Identify the user sending the excessive queries in vpxd.log. Refactor the custom application so that it makes less frequent auth queries to vCenter.

You can use grep "GetUserInfoInternal" vpxd.log    to get all "GetUserInfoInternal" calls for user accounts, and then by working with cut, uniq, and wc, you can strip the usernames out specifically, filter to distinct users, and then get a count of lines that make the GetUserInfoInternal call with that user account.

You can also use grep "<username>" vpxd-profiler* | grep -i "clientip" to get a list of IP addresses where the user calls are coming from.