"Error: tls: failed to find any PEM data in key input" on Supervisor ETCD node
Article ID: 448968
Updated On:
Products
Issue/Introduction
TLS-related errors are observed in the Supervisor ETCD logs.
Checking node status shows that one of the control plane nodes is in the NotReady state:
# kubectl get nodes
NAME STATUS ROLES
423d0a7caad36bdfc6154f32cabc09e9 NotReady control-plane,master
423d16742a524e2bb7d43039c8c54515 Ready control-plane,master
423dcf3ecaaac3d319a5c66d6c64e5f4 Ready control-plane,master
The following error is observed on the Supervisor node in the NotReady state:
# etcdctl endpoint health --cluster -wtable
Error: tls: failed to find any PEM data in key input
Checking cluster health from another healthy Supervisor node shows that the affected node is lost as an ETCD member:
# etcdctl endpoint status --cluster -wtable
{"level":"warn","ts":"YYYY-MM-DDThh:mm:ss.ssssZ","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004301e0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp #.#.#.#:2379: connect: connection refused\""}
Failed to get the status of endpoint https://#.#.#.#:2379 (context deadline exceeded) +----------------------+------------------+----------------------+---------+-----------+-----+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | ... | +----------------------+------------------+----------------------+---------+-----------+-----+ | https://#.#.#.#:2379 | 33366e14b5235488 | 3.5.21+vmware.1-fips | 102 MB | false | ... | | https://#.#.#.#:2379 | 5dea4a0560086169 | 3.5.21+vmware.1-fips | 103 MB | true | ... | +----------------------+------------------+----------------------+---------+-----------+-----+
Environment
vSphere Kuberentes Service - Supervisor
Cause
This issue occurs if an OS-level failure (such as a disk full condition) occurs during automatic certificate renewal, causing the contents of the ETCD TLS keys to be lost.
As a result, the ETCD TLS key files become empty (0 bytes):
# ls -l /dev/shm/wcp_decrypted_data
-rw------- 1 etcd etcd 0 MM DD hh:mm etcd-ca-key
-rw------- 1 etcd etcd 0 MM DD hh:mm etcd-healthcheck-client-key
-rw------- 1 etcd etcd 0 MM DD hh:mm etcd-peer-key
-rw------- 1 etcd etcd 0 MM DD hh:mm etcd-server-key
-rw------- 1 root root 0 MM DD hh:mm k8s-etcd-client-key
Resolution
Perform the following steps on the affected Supervisor node in the NotReady state.
1. Stop the kubelet, etcd, and kube-apiserver services.
systemctl stop kubelet
crictl stop $(crictl ps -q --name etcd)
crictl stop $(crictl ps -q --name kube-apiserver)
crictl ps | grep -E '(kube-apiserver|etcd)' # (no output)
2. Restart the hypercrypt service and start kubelet.
systemctl restart hypercrypt
systemctl start kubelet
crictl ps | grep -E '(kube-apiserver|etcd)' # Running
3. Confirmation
# Confirm that the key files are restored (non-zero file size):
ls -l /dev/shm/wcp_decrypted_data
# Verify that all 3 nodes are healthy:
etcdctl member list -wtable
etcdctl endpoint status --cluster -wtable
etcdctl endpoint health --cluster -wtable
kubectl get nodes -owide
Additional Information
Feedback
