Non-Master NSX Manager Node Unable to Connect to vCenter Due to DNS Resolution Failure

Products

• VMware NSX

Issue/Introduction

A non-master NSX Manager node fails to establish a connection with vCenter Server due to a DNS resolution issue.
Common scenarios where this issue may occur:
1. After running the CARR script to replace expiring certificates, multiple transport nodes lose controller connectivity
2. After redeploying an NSX Manager node without proper DNS configuration for vCenter

Symptoms

• In NSX Manager logs, an exception occurs during communication between NSX Manager and vCenter.
  Log Location: /var/log/proton/nsxapi.log
  com.vmware.nsx.management.switching.common.exceptions.SwitchingException: com.vmware.vim.vmomi.client.exception.ConnectionException: https://sdkTunnel:8089/sdk/vimService invocation failed with "java.net.UnknownHostException: test-host.test.local"
      at com.vmware.nsx.management.switching.infrastructure.service.L2ManagedObjectOwnershipManagerImpl.stampDVSOwnership(L2ManagedObjectOwnershipManagerImpl.java:139)
      at com.vmware.nsx.management.switching.infrastructure.service.L2ManagedObjectsOwnershipMgmtServiceImpl.stampDvsOwnership(L2ManagedObjectsOwnershipMgmtServiceImpl.java:132)
      at com.vmware.nsx.management.switching.sync.host.TransportNodeAsyncServiceImpl.processNsxOwnershipForTransportNode(TransportNodeAsyncServiceImpl.java:3394)
      at com.vmware.nsx.management.switching.sync.host.TransportNodeAsyncServiceImpl.syncTransportNode(TransportNodeAsyncServiceImpl.java:3191)
      ...
  Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: https://sdkTunnel:8089/sdk/vimService invocation failed with "java.net.UnknownHostException: test-host.test.local"
      at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:265)
      at com.vmware.nsx.management.inventory.access.vc.VcConnectionImpl.loginByCertificateAsExtensionOneCall(VcConnectionImpl.java:471)
      at com.vmware.nsx.management.inventory.access.vc.VcConnectionManagerImpl.getVcConnection(VcConnectionManagerImpl.java:237)
      at com.vmware.nsx.management.inventory.access.vc.VcOperationManagerImpl.setNsxOwnership(VcOperationManagerImpl.java:56)
• The Compute Manager status is UP and REGISTERED. This confirms the DNS issue further, as the compute manager is healthy

Environment

VMware NSX 

Cause

The NSX Manager node is unable to resolve the vCenter FQDN, which impacts connectivity between the NSX Manager and vCenter.

Resolution

1. Run nslookup "<vcenter-fqdn>" on all NSX Manager nodes to verify DNS resolution
2. If resolution fails, update the DNS server configuration on the affected node(s)

Workaround:
Uninstall NSX on the affected transport node and re-install NSX.

Additional Information

CARR script to replace the certificates: https://knowledge.broadcom.com/external/article/369034