The vCenter Server connects to the Secondary DNS server even if the Primary DNS Server is reachable.

Products

• VMware vCenter Server

Issue/Introduction

In a situation where vCenter has 2 DNS servers which are not in sync, domain logins to the vCenter server might fail for entries not present in the 2nd DNS server.
This is due to vCenter reaching out to the secondary DNS even though the primary DNS server is available.

Below snippet is observed in websso logs during a failed AD login:
"cannot establish ldap connection with URI: [ldap://<server_name>:389] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use secondary URIs, if applicable".

This issue can cause intermittent AD User logins failure with an "invalid credentials" error.

Environment

vCenter Server 7.x

vCenter Server 8.x

 

Cause

• On vCenter server, DNS queries are handled by systemd-resolved service.
• It follows internal rules to choose the external DNS server.
• This might be different for each query made by the vCenter Server.
• It uses information based on if a server worked or failed in the past, interface order and even what domains are allocated to each interface.
• In most scenarios, it will select a DNS server based on which DNS responds first, irrespective of which Server is defined as the primary server.

Resolution

Add a flag to force the systemd-resolved service to prefer a particular DNS server.
This flag forces dnsmasq to try each query with each server strictly in the order they appear in /etc/resolv.conf.
dnsmasq will only refer to the next server in the order only if the preferred one is unreachable.

To force the vCenter Server to always query the Primary DNS server if it is reachable:

1. SSH to the vCenter Server as root.
   
   
2. vi /etc/dnsmasq.conf
   
   
3. Add the strict-order string (in bold)  after the log-queries string:

     listen-address=127.0.0.1
      bind-interfaces
      user=dnsmasq
      group=dnsmasq
      no-negcache
      no-hosts
      log-queries
      strict-order
      log-facility=/var/log/vmware/dnsmasq.log
      domain-needed
      dns-forward-max=150
      cache-size=8192
      neg-ttl=3600

4. Restart dnsmasq service to apply the changes:
   systemctl daemon-reload
systemctl restart dnsmasq.service

Additional Information

This configuration is helpful in environments where some DNS entries are not defined in the Secondary Server, usually where the DNS Servers do not synchronize with each other.

Making the above changes makes sure that the queries for the entries are not forwarded to the secondary servers causing login failures for those accounts.