kubectl commands fail against a Kubernetes cluster in Tanzu Kubernetes Grid Integrated Edition

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

• You see that kubectl commands connecting to the Kubernetes cluster API fail without any error. 
• You see messages similar to the following when appending the -v 5 (verbosity level five) to your kubectl commands:

I1028 16:29:44.055331   35366 request.go:784] Got a Retry-After 1s response for attempt 1 to https://<API-URL-IP>:8443/api?timeout=32s
I1028 16:29:45.079933   35366 request.go:784] Got a Retry-After 1s response for attempt 2 to https://<API-URL-IP>:8443/api?timeout=32s

• You see messages similar to the following in the kub-api logs after each attempted command:

I1028 16:29:45.260105       9 log.go:172] http: TLS handshake error from 192.###.#.##:9398: read tcp 172.##.##.#:8443->192.###.#.##:9398: read: connection reset by peer

• You see messages similar to the following when attempting a curl command (with verbose output, -v) to the Kubernetes API endpoint:

* About to connect() to <API-URL-IP> port 8443 (#0)
*   Trying ###.###.###.###...                            --------------------------> External API Virtual IP Address
* Connected to <API-URL-IP> (10.##.##.#) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer

Resolution

• This issue can occur when the TCP connection is allowed but when data is passed to establish the TLS handshake, the connection is denied. The TCP reset RST is commonly what gets sent back by a Deep Packet Inspection DPI type firewall when the firewall determines the data should be denied based on the TLS handshake that was sent.
• NSX-T does have a layer 7 firewall ability, however this is not configured by default. Many other firewalls also can have DPI features.
• If the traffic is going through a NSX-T firewall, check if there are any Layer 7 Context Profiles configured that would be blocking traffic. Details on layer 7 context profiles and firewall rules can be found in Layer 7 Context Profile.
• If NSX-T does not have a layer 7 firewall rule blocking, check the external firewalls for DPI rules that could be blocking traffic. 

 

Workaround:

Run kubectl commands from the master node locally. (this is a stop gap measure if the firewall issue cannot be immediately addressed.)

1. 1. Get the bosh cli credentials from the OpsMan UI in the bosh tile credentials tab.
   2. SSH to the OpsMan VM.
   3. Run bosh vms to find the deployment ID. The deployment ID matches the cluster ID from the TKGI cli.
   4. Run bosh -d <deployment-id> ssh master/0 to ssh to the master node.
   5. Switch to root by running sudo su -
   6. Add the location to the kubernetes executable  to the env path by running export PATH="${PATH}":/var/vcap/packages/kubernetes/bin/
   7. Run kubectl commands as you normally would.

Note: If you need to use specific credentials, you will need to copy your local .kube/config to ~./kube/config on the master node.