Error "Transport: Error while dialing: error upgrading connection: error sending request:" observed when control plane and worker nodes get into a restart loop.

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Control plane and worker nodes are unexpectedly restarting and entering an automated redeployment cycle.
• Affected nodes are observed transitioning into a Ready, SchedulingDisabled state.
• Cluster events and system logs indicate that these nodes are continuously failing to reach the configured floating IP address.

capi logs

2024-04-14T16:29:23.99761333Z stderr F {"level":"warn","ts":"2024-04-14T16:29:23.997259Z","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005f183c0/etcd-tkg-controlplane-8","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: error upgrading connection: error sending request: Post \\\"https://172.x.0.0:6443/api/v1/namespaces/kube-system/pods/etcd-tkg--controlplane-/portforward?timeout=10s\\\": dial tcp 172.x.0.0:6443: connect: connection refused\""}

etcd logs

2024-04-14T16:29:23.99761333Z stderr F {"level":"warn","ts":"2024-04-14T16:29:23.99761333Z ","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2
","local-member-id":"######","remote-peer-id":"xxxxxxxx","error":"EOF"} 
2024-04-14T16:29:23.99761333Z  stderr F {"level":"warn","ts":"2024-04-14T16:29:23.99761333Z ","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message",

Environment

TKG 2.5.1

Cause

An underlying network infrastructure routing or connectivity failure prevents the control plane and worker nodes from communicating with the floating IP address, which causes health checks to fail and triggers the automated node redeployment cycle.

Resolution

1. Verify network routing and firewall policies between the affected cluster nodes and the floating IP address subnet.

2. Engage the internal network infrastructure team to investigate and restore network reachability to the floating IP address.

3. Validate connectivity from the node network using standard diagnostic tools.

4. Monitor cluster health and system events to confirm nodes successfully connect to the floating IP address and stabilize in a Ready state with scheduling enabled.