vmdir replication between the vCenter is broken due to DNS issue getting "Could not connect to one or more vCenter Server Systems: https://<vCenter-FQDN-Or-IP>:443/sdk" error.

Products

• VMware vCenter Server

Issue/Introduction

• We will observe a banner in the vSphere UI with error,

  • Could not connect to one or more vCenter Server Systems: https://<vCenter-FQDN-Or-IP>:443/sdk

  • Authentication failed for  one or more vCenter Server Systems: https://<vCenter-FQDN-Or-IP>:443/sdk

• vmdir replication between the vCenter is broken. 
• • From vCenter, you will not be able to get the status of the replication status. 
  • From partner vCenter, you will observe ## change behind. 
• Rebooting or restarting vCenter services take longer than expected to reach the running state.
• If we check the vmdir state, it is Normal state from all the vCenter in Enhanced Linked Mode.

  • root@vCenter [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli state get
    Enter password for [email protected]:
    Directory Server State: Normal (3)

• From /var/log/vmware/vmdird/vmdird.log, we will observe similar events,

  • YYYY-MM-DDTHH:MM:SSZ  ERROR: VmDirAnonymousLDAPBindEx to (ldap://<Partner_vCenter_FQDN>:389) failed. (-1)(Can't contact LDAP server)
    YYYY-MM-DDTHH:MM:SSZ  ERROR: VmDirGetServerName failed with error (9127)
    

• If we check the port connectivity over 389 between vCenter using FQDN it fails and with IP address it works.

  • curl -v https://<Partner_vCenter_FQDN>:389

• Also, ESXi host in the vCenter inventory will be in not responding state (If hosts are added to the vCenter with FQDN) 

Environment

• VMware vCenter Server

Cause

• The problem was caused by the unavailability of the configured DNS server(s). This can prevent proper hostname resolution for the vCenter Server Appliance (VCSA) and other dependent services.
• Check resolving all DNS Server as well as VCSA FQDN using nslookup command 
  

  • nslookup <VCSA_FQDN>
    nslookup <DNS_FQDN>

• Also, run the below command to verify port connectivity to DNS server. 

  • curl -v telnet://<DNS_IP_ADDRESS>:53

Resolution

• Verify DNS Server Reachability:
  • Ensure the DNS server is reachable from the vCenter Server.
  • Use network utilities such as ping or nslookup to confirm connectivity and name resolution.
    
    
• Check Multiple DNS Servers:
  • If multiple DNS servers are configured, verify that all are operational and healthy.
  • Remove or replace any non-responsive DNS server entries to avoid resolution failures.
    
    
• Configure Healthy DNS Server in vCenter:
  • Confirm that at least one healthy DNS server’s IP address is correctly configured in the vCenter Server network settings.
  • Update the configuration if the current DNS entries point to unhealthy or unreachable servers.

Additional Information

• DNS Requirements for the vCenter Server Appliance
• To add or modify DNS configuration on vCenter, please refer the below article.
  • How to change/update DNS Server IP address for vCenter Server