Supervisor Cluster has the error "Error: error occurred when attempting to connect to NSX Manager"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In wcpsvc.log, we notice similar errors where the communication to apiserver failed

2###-##-##T##:##:##.269Z warning wcp [kubelib/retry.go:93] [opID=6824fb01] Request to apiserver failed. Err , Endpoint http://localhost:1080/external-cert/http1/x.x.x.255/6443/version?timeout=2m0s. Will be retried.

There may be customized guest of Supervisor Control plane VM failures in the vSphere UI.

vSphere Workload Management UI may show the following components stuck configuring

• Configured Supervisor Control plane VM as Kubernetes Control Plane Node
  
  
• Configured Supervisor Control plane VM's Workload Network
  • Configuration error (since <timestamp>)
  • Unable to connect to NSX Manager (https://nsx.example.com:443) from control plane VM 4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx3. Error: error occurred when attempting to connect to NSX Manager. err: Get "https://nsx.example.com:443/": error connecting with TLS: error directly dialing with certificate in config: dial tcp: lookup nsx.example.com:443 on 127.0.0.53:53: read udp 127.0.0.1:40707->127.0.0.53:53: i/o timeout.

Environment

vCenter Server 8.x
vCenter Server 7.x

Cause

This happens if the Supervisor Control Plane VM's cannot communicate to the NSX Manager. 

Resolution

Validate that the networking between the Supervisor Control Plane VM's (SV VMs) and the NSX Manager is functional. 

In order to test this, first SSH into the SV VM's via this KB: https://knowledge.broadcom.com/external/article?legacyId=90194 

From Supervisor control plane VM check the following. 

• ping vCenterFQDN/IP
• ping NSXManagerFQDN/IP
• Test the DNS using dig/nslookup
  • dig vCenterFQDN/IP
  • dig NSXManagerFQDN/IP
  • nslookup vCenterFQDN/IP
  • nslookup NSXManagerFQDN/IP
• Validate the port communication over port 443 using openssl and curl
  • openssl s_client -connect <vCenterFQDN>:443
  • openssl s_client -connect <NSXMGRFQDN/VIP>:443
  • curl -vvv https://vCenterFQDN
  • curl -vvv https://NSXFQDN

*Please note that if NSX is configured with 3 NSX-Managers to test all 3 + the NSX Manager VIP as part of troubleshooting. Check the wcpsvc.log and the UI to see what specific IP it is trying to use however.