Supervisor Cluster Unable to connect to the management DNS servers from control plane VM - The connection was attempted over the workload network

Products

VMware vSphere Kubernetes Service VMware vSphere 7.0 with Tanzu vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

In the vSphere web client under Workload Management for Supervisors, the Supervisor Cluster shows one or more Errors.

When clicking on the Error number count in parentheses (), the following error messages are present, where the values in brackets <> will vary by environment:

Unable to connect to the management DNS servers '<DNS server(s)>' from the control plane VM <Supervisor VM DNS name>.
Configured Supervisor Control plane VM as Kubernetes Control Plane Node
Configured Supervisor Control plane VM's Workload Network
 • Configuration error (since MM/DD/YYYY, HH:MM:SS)
 • Unable to connect to NSX Manager (https://<NSX Manager FQDN>:443) from control plane VM <Supervisor VM DNS name>. Error: error occurred when attempting to connect to NSX Manager. err: Get "https://<NSX Manager FQDN>:443/": error connecting with TLS: error directly dialing with certificate in config: dial tcp: lookup <NSX Manager FQDN> on 127.0.0.53:53: read udp 127.0.0.1:#####->127.0.0.53:53: i/o timeout.

The DNS servers configured in the Supervisor cluster's Management Network and Workload Network are the same.

This can be found under the Supervisor inventory object > Configure > Network

While SSH into one of the Supervisor Cluster control plane vms, the DNS server(s), NSX FQDN and vCenter FQDN cannot be reached:

See "How to SSH into Supervisor Control Plane VMs" from Troubleshooting vSphere with Tanzu Supervisor Control Plane VMs
- ```
nslookup <dns server>
```
- ```
curl -vk <vcenter FQDN>
```
- ```
nslookup <vcenter FQDN> <dns server>
```
- ```
curl -vk <nsx manager FQDN>
```

For environments using the NSX load balancer, the nsx-ncp pod is in CrashLoopBackOff state:

kubectl get pods -n vmware-system-nsx

The logs of nsx-ncp pod show errors similar to the below that it cannot reach the vCenter over its FQDN:

kubectl logs -n vmware-system-nsx <nsx-ncp pod name> -c nsx-ncp

stderr F [ncp MainThread W] nsx_ujo.ncp.vc.session Failed to get JWT token: Failed SAML HoK request: Failed to get or renew SAML HoK from STS due to failed DNS
 lookup for VC endpoint: [Errno -3] Lookup timed out., will retry after 120 seconds

kubectl logs -n vmware-system-nsx <nsx-ncp pod name> -c nsx-operator

failed to create VIM client     {"vimSdkURL": "https://<vCenter FQDN>:443/sdk", "error": "Post \"<vCenter FQDN>:443/sdk\": dial tcp: lookup <vCenter FQDN> on 127.0.0.53:53: server misbehaving"}
JWT cache failed to refresh JWT         {"error": "Post \"https://<vCenter FQDN>:443/sdk\": dial tcp: lookup <vCenter FQDN> on 127.0.0.53:53: server misbehaving"}

This same issue will also result in all Workload clusters managed by the affected Supervisor cluster failing to create new nodes/VMs and reporting similar lookup failures.

Environment

vSphere Supervisor

Cause

One or more Supervisor cluster control plane virtual machines are unable to reach the DNS servers configured in the environment, specifically over the Workload Network.

This could be due to a networking configuration issue, a change to the DNS server configuration or an issue with the DNS service on the affected Supervisor cluster control plane vm(s).

If the Workload Network DNS servers are the same as the Management Network's DNS servers, DNS traffic is routed through the Workload Network.

Resolution

The connection between the affected Supervisor control plane VM(s) and the DNS server(s) will need to be fixed.

Check through the list detailed in the below documentation:
- Resolving Errors Health Statuses on Supervisor Cluster During Initial Configuration or Upgrade (vSphere 8)
Connect into the affected Supervisor control plane VM
- See "How to SSH into Supervisor Control Plane VMs" from Troubleshooting vSphere with Tanzu Supervisor Control Plane VMs
Confirm on the status of systemd-resolved and that it is using the correct DNS servers:
```
systemctl status systemd-resolved

systemd-resolve --status
```
Check that the Supervisor VM's resolv.conf file is not modified from the default shown below and that it has the correct, functional DNS Search Domain(s):
```
cat /etc/resolv.conf

nameserver 127.0.0.53
options ends0 trust-ad
search <dns search domain configured for the Supervisor cluster's management network>
```
If the DNS server configuration needs to be updated, see the following documentation:
- Configure DNS, IP Address, and Proxy Settings for the vCenter Server Appliance (VCSA)
- Modify the List of Supervisor Control Plane DNS Names
- After any DNS changes to the Supervisor Cluster, confirm that these are reflected in systemd-resolved:
```
systemctl status systemd-resolved

systemd-resolve --status
```
If all of the DNS configuration as per the above steps is correct, but the issue persists, involve your networking team and networking experts to troubleshoot the DNS connection failures from all Supervisor VMs over both NICs (eth0, eth1).
- All Supervisor VMs use two NICs, one for the management network and one for the workload network.
- DNS must work over both the management and workload network for all system components and services to function properly.

Additional Information

Supervisor enablement fails with DNS server connection timeout error

In VCF 9.x, DNS traffic is routed through the Management Network regardless of the DNS servers being identical on both the Workload Network and Management Network.