PV mount fails with "tls: failed to verify certificate: x509: certificate signed by unknown authority"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Clusters having problems attaching PV in any guest cluster

AttachVolume.Attach failed for volume "pvc-########-####-####-####-############" : rpc error: code = Internal desc = failed to get VirtualMachines for the node: "<node name>". Error: conversion webhook for vmoperator.vmware.com/v1alpha2, Kind=VirtualMachine failed: Post "https://vmware-system-vmop-webhook-service.vmware-system-vmop.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority

VMOP logs show the following error

E0410 18:10:29.691290       1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1alpha2.VirtualMachineService: failed to list *v1alpha2.VirtualMachineService: conversion webhook for vmoperator.vmware.com/v1alpha1, Kind=VirtualMachineService failed: Post "https://vmware-system-vmop-webhook-service.vmware-system-vmop.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "serial:###################################")

Certificate is valid

kubectl get secret webhook-server-cert -n vmware-system-vmop -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -text

Environment

• vSphere with Tanzu

Cause

There are many "certificate signed by unknown authority" and "bad certificate" errors in various pods logs, but no "certificate has expired" errors.

Resolution

Restart the Cert Manager Cainjector deployment

kubectl rollout restart deployment -n vmware-system-cert-manager cert-manager-cainjector