Etcd Logs Report "tls: client didn't provide a certificate" in vSphere Kubernetes Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• In VMware vSphere Kubernetes cluster, etcd appears healthy, but etcd and kubelet logs report frequent warnings regarding rejected incoming connections.
• In etcd container logs below warnings are observed every few minutes:
  
  crictl logs <etcd container_id>
  
  {"level":"warn","caller":"embed/config_logging.go:###","msg":"rejected connection on client endpoint","remote-addr":"<IP>:<PORT>","server-name":"","error":"tls: client didn't provide a certificate"}
  
  
• Kubelet Logs on Control plane node show below warnings:
  
  journalctl -xeu kubelet
  
  YYYY-MM-DD:T:HH:MM:SS <Control plane Node> kubelet[#####]: I### http: TLS handshake error from <IP>:<PORT>: client sent an HTTP request to an HTTPS server"
"############ upgradeaware.go:#####] Error proxying data from backend to client: write tcp  <IP>:<PORT> ->  <IP>:<PORT> : write: broken pipe"
YYYY-MM-DD:T:HH:MM:SS <Control plane Node> kubelet[#####]: I###  upgradeaware.go: Error proxying data from backend to client: writeto tcp  <IP>:<PORT> ->  <IP>:<PORT> write tcp <IP>:<PORT> ->  <IP>:<PORT> write: broken pipe
  
  

Environment

VMware vSphere Kubernetes Service

Cause

An application residing on a worker node (commonly a monitoring tool) is polling the etcd client endpoint on port 2379 using HTTPS or HTTP, but it is not providing the required Mutual TLS (mTLS) client certificates. The etcd server rejects the connection because peer verification fails.

Resolution

To resolve the alerts, identify the source application and update its configuration with valid certificates.

1. Identify the Source Pod: SSH into the control plane node and search for the source IP identified in the logs:
   
   kubectl get pods -o wide -A | grep -E "<SOURCE_IP>"
   
   
2. Confirm Connection Attempts: SSH into the worker node hosting the pod identified in the previous step. Run a packet capture to confirm the application is attempting connections to the etcd control plane on port 2379: 
   
   sudo tcpdump -i any -nn -c 20 'dst port 2379 and (tcp[tcpflags] & tcp-syn != 0)'


3. Update Application Configuration: Update the configuration of the identified application to include valid etcd client certificates.

Note: The etcd certificates (.ca, .crt, and key) are located on the control plane node at: /etc/kubernetes/pki/etcd/