Pod age varies after restart.

Products

• vSphere with Tanzu
• VMware vSphere Kubernetes Service

Issue/Introduction

• Pod age varies after restart inside the Guest cluster.

Example: From the ssh of the Guest cluster Control plane, after we delete the pod 'test-pod-##########-####', the age of the pod after re-deployment is not even:

 ~ k get pods
NAME                                  READY   STATUS    RESTARTS   AGE
test-pod-##########-####   1/1     Running   0          3h2m

~ k delete pod test-pod-##########-####;date
pod "test-pod-##########-####" deleted
Wed Jan 22 15:40:04 IST 2025

~ kubectl get pods -n testns;date
NAME                                  READY   STATUS    RESTARTS   AGE
test-pod-##########-####   1/1     Running   0          6s
Wed Jan 22 15:40:09 IST 2025

~ kubectl get pods -n testns;date
NAME                                  READY   STATUS    RESTARTS   AGE
test-pod-##########-####   1/1     Running   0          9s
Wed Jan 22 15:40:12 IST 2025

~ kubectl get pods -n testns;date
NAME                                  READY   STATUS    RESTARTS   AGE
test-pod-##########-####   1/1     Running   0         3m7s
Wed Jan 22 15:40:21 IST 2025

~ kubectl get pods -n testns;date
NAME                                  READY   STATUS    RESTARTS   AGE
test-pod-##########-####   1/1     Running   0         22s
Wed Jan 22 15:40:25 IST 2025

• Issue with system as well as non system pods and also with namespaces created within the guest cluster.
• When running timedatectl command on the Guest cluster nodes it shows:

System clock synchronized: no

• Service systemd-timesyncd inside the Guest cluster nodes shows running.

systemctl status systemd-timesyncd

Environment

vSphere with Tanzu

Cause

This is a known issue caused due to systemd-timesyncd service resulting in time drift across the control plane nodes as well as worker nodes.

Resolution

• The issue is fixed in 1.32 TKr release. Please refer TKr v1.32.0 for vSphere 8.x.
• To workaround the issue, disable the systemd-timesyncd service on all the nodes of that Guest cluster having the issue.
  
  
  • Stop service:

systemctl stop systemd-timesyncd  

• • Disable service:

systemctl disable systemd-timesyncd