Failed to create service in VKS guest cluster due to expired certificate

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Creating service in VKS guest cluster, executing command "kubectl describe svc <service name> -n <namespace>" to describe service, there is warning event with "SyncLoadBalancerFailed" as reason and "Error syncing load balancer: failed to ensure load balancer: VirtualMachineService IP not found" as Message:
  
  
  
  
  
• In vmware-system-vmop-controller-manager pod log, there is error like below:
  
  YYYY-MM-DDTHH:MM:SSZ stderr F EMMDD HH:MM:SS.XXXXX        X virtualmachineservice_controller.go:xxx] VirtualMachineService "msg"="Failed to update VirtualMachineService k8s Service" "error"="Internal error occurred: failed calling webhook \"admission.vmware.com\": Post \"https://xxx.xxx.xxx.xxx:xxxxx/scheduler/admission?timeout=10s\": x509: certificate has expired or is not yet valid: current time YYYY-MM-DDTHH:MM:SSZ is after YYYY-MM-DDTHH:MM:SSZ" "name"="<VKS namespace>/<VKS node name>"


Environment

VMware vSphere Kubernetes Service

Cause

Expired vSphere supervisor certificate led to this issue.

Resolution

Use certmgr to check and rotate vSphere supervisor certificates:

Replace vSphere Supervisor (Previously known as vSphere with Tanzu) Certificates