Troubleshooting authentication failures with Active Directory users in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

This article provides guidance when troubleshooting issues related to Active Directory authentication in vCenter Server. 

• Authentication to vCenter with Active Directory credentials is slow or fails intermittently
• vCenter - /var/log/vmware/messages contains many of the following errors:
  
  Failed to find user, group, or domain by name (name = '<AD_USER>', searched host = '<DOMAIN_CONTROLLER>') -> error = 40098, symbol = LW_ERROR_RPC_OPENPOLICY_FAILED

Detected domain 'domain.example' offline. Some group information from this domain might be missing

DNS lookup for '_ldap._tcp.dc._msdcs.domain.example' failed with errno 110 (Connection timed out)
  
  
• vCenter - /var/log/vmware/messages contains many of the following errors during the incident:
  
  lwsmd: Restarting dead service: netlogon (attempt 1/2)

Environment

VMware vCenter Server

Cause

The root cause of failed Active Directory logins can come from several sources such as:

• Excessive login attempts by another solution or external script
• Poor performance of queries to Active Directory
• vCenter Appliance is running low on resources
• A known issue with netlogond crashes (fixed in vCenter Server 6.7 U3)

Note: Critical security hardening measures were introduced in vCenter Server 6.7 which affect Kerberos authentication performance.

Resolution

Use the following methods to determine the cause of AD Authentication issues.

• Excessive Logins
• Problems Communicating with Active Directory
• vCenter Appliance is Running Low on Resources
• Netlogond Crashes
• Workaround

Excessive Logins

Login attempts are typically recorded in log files or the system journal and will looks similar to the example below.

• API Authentication:  vCenter - /var/log/vmware/sso/vmware-identity-sts.log
• Web Client Authentication:  vCenter - /var/log/vmware/sso/websso.log
  
  [2020-03-18T17:20:02.958Z tomcat-http--11 vsphere.local        b9a75dd5-0ae6-4304-8dcf-885ac1a04bf2 INFO  com.vmware.identity.idm.server.IdentityManager] Authentication succeeded for user [[e-mail]] in tenant [vsphere.local] in [131] milliseconds with provider [DOMAIN.EXAMPLE] of type [com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider]

• journalctl -b (/ then allows for searching for the offending username)
  
  Example: [error] [USER]  [ ]  [1409351611]  [Cannot login USER@##.###.###.##]

The log entry above shows several pieces of information about the authentication attempt.

• Whether the authentication succeeded or failed
• The username
• How long the authentication attempt took
• The domain the user belongs to

Defining an upper limit to these attempts is difficult because it depends on the latency between the likewise service on the VCSA and the domain controller responses. If there are several logins coming from a single source per minute, it indicates improper session use practices by that solution or product.

Excessive logins must be addressed at their source, usually with a patch or workaround.

Problems Communicating with Active Directory

Problems communicating with domain controllers from trusted or child domains is known to cause AD authentication problems. Failed queries to these domains can build up and cause likewise to eventually crash. The following errors are an indication that a trusted domain is not responding.

• vCenter - /var/log/vmware/messages

Detected domain 'domain.example' offline. Some group information from this domain might be missing

AD communication issues may also be related to DNS. vCenter requires forward and reverse resolution to work properly on all domain controllers it communicates with.

• DNS related issues can be detected with the checkADConfig.sh script
• For information on how to use the checkAdConfig.sh script, see KB: Using checkADConfig to detect connectivity and DNS issues between vCenter Server and Active Directory 

Even without excessive logins, it’s still possible for likewise to become overwhelmed when there is a high latency between vCenter and Active Directory. Highly latent environments will cause authentication requests to queue up in likewise and eventually fail.

• Latency to Active Directory can be detected with the lw_measure tool.
• For information on how to use the lw_measure tool, see KB: Using the lw-measure tool to detect latency between vCenter Server and Active Directory domain controllers 

vCenter Appliance is Running Low on Resources

If the vCenter Appliance is under-sized for its environmental load, it can cause authentication to fail among other issues.

• vCenter Server Appliance performance can be monitored using vimtop.
• For more information about vimtop, see Using the vimtop Plug-In to Monitor the Resource Use of Services
   

Netlogond Crashes

The netlogon daemon is responsible for communication with the domain controllers. A known issue with netlogond crashes has been resolved in vCenter Server 6.7 Update 3.
When netlogond crashes and is restarted by the likewise service manager, it will print entries similar to below.

• vCenter - /var/log/vmware/messages
  
  lwsmd: Restarting dead service: netlogon (attempt 1/2)
lwsmd: Service died: netlogon (restarted 2 times in 12 seconds)

Workaround

Creating and utilizing an SSO domain service account can alleviate some of the performance impact for products or solutions that authenticate to vCenter with Active Directory credentials. See Managing vCenter Single Sign-On Users and Groups for more information

Configuring the vCenter Server to use AD over LDAPS as an identity source will also help alleviate issues pertaining to excessive logins and likewise related delays. See Active Directory LDAP Server and OpenLDAP Server Identity Source Settings for more information.