After upgrading to vCenter, PVCs cannot be created, or when a pod is restarted, they fail to connect to the attached pod

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When a Pod is created or restarted it's unable to attach to its PVC. A Pod could be restarted due to a cluster upgrade, a deployment being modified, etc
• Velero is installed as a Supervisor service.
• CAPV pods show the following error:
  

  Internal error occurred: failed calling "webhook cbt.mutate.virtualmachine": failed to call webhook: Post: "https://velero-vsphere-webhook-service.svc-velero-vsphere-domain-####.svc:443/cbt-mutate-vmop-vm?timeout=10s": dial <IP_ADDRESS>:443: connect: connection refused.

• CSI vSphere Controller Pods shows the following error:
  

  CNS CreateVolume failed from vCenter "<VCENTER_URL>"

• Velero Pods are failing to start with the error:
  

  Error creating: pods "velero-vsphere-operator-####-###" is forbidden: violates PodSecurity "baseline:latest" host namespaces (hostNetwork=true)

Environment

vSphere with Tanzu 8.0

Cause

The Pod Security Policy in the Velero Kubernetes namespace is preventing the creation of the Velero Pods. When a Pod attempts to create to a PVC, the PVC needs to be attached to a Kubernetes cluster worker node. This requires that the PVC attachment request goes through the Velero mutating webhook, which talks to the Velero service running in the Velero Pods. Since the Pods can't start due to the PodSecurity policy, the request fails with a timeout error.

Resolution

The Pod Security Policy in the Velero namespace needs to be modified to allow the Pods to start. Reference Configure PSA for TKR 1.25 and Later for more information on configuring Pod Security Policies.

Example Pod Security Policy:

kubectl label --overwrite ns <VELERO_NAMESPACE> pod-security.kubernetes.io/enforce=privileged