Error “watch chan error: etcdserver: mvcc: required revision has been compacted” is observed in kube-apiserver

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Kube-api server logs show a spew of “watch chan error: etcdserver: mvcc: required revision has been compacted”

Environment

2.x

Cause

• This error is a common occurrence in Kubernetes or systems using etcd as their key-value store. It essentially means that a client (like kube-apiserver) is trying to "watch" or read a specific version of data that etcd has already deleted to save space.
• Etcd keeps a history of all changes (revisions). To prevent the database from growing indefinitely, etcd performs compaction, which permanently removes old revisions of keys.
• If this error message appears for a small duration in the logs and then stops, the system recovered itself.
• But it’s looping, a specific pod or controller might be stuck and needs a restart.

Resolution

Option A: 

Determine the current revision number and delete the older revisions

1. Identify the current revision
   
   etcdctl endpoint status -w json | jq '.[].Status.header.revision'
   
   
2. Compact the History (Using the revision number from previous output)
   
   etcdctl compact <REVISION_NUMBER>
   
   
3. Defragment the Database (Note: his is a blocking operation. While a node is defragmenting, it will not respond to requests. In a 3-node cluster, run this one node at a time)
   
   etcdctl defrag
   
   

Option B

Identify if there is a specific pod which is causing this error spew and delete the pod.

1. Run the below command to check if there is a specific pod which is generating this error
   
   kubectl get pods -A --no-headers -o custom-columns=":metadata.namespace,:metadata.name" | xargs -n2 sh -c 'kubectl logs -n $0 $1 --tail=100 2>/dev/null | grep -i "compacted" && echo "Found in: $0/$1"'
   
   
2. Query the kubeserver-api log and confirm that the pod is generating the error
   
   kubectl logs -n kube-system -l component=kube-apiserver --tail=1000 | grep -i "compacted"
   
   
3. Delete the pod 
   
   kubectl delete pod <pod-name> -n <namespace>

This will clear the local cache of the old resourceVersion and perform a fresh LIST request to etcd to receive the latest revision, and start a new WATCH.