CNS error failed to add task to list view. err: failure to add task to listview. task: Task:task-<task id>, err: ServerFaultCode: The session is not authenticated.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• PVC creation fails with "failed to provision volume with StorageClass "<storage class name>": rpc error: code = Internal desc = failed to create volume. Error: failed to add task to list view. err: failure to add task to listview. task: Task:task-<task id>, err: ServerFaultCode: The session is not authenticated."
• Describe of pvc shows below error:
  kubectl describe pvc <pvc name> -n <namespace>

Events:
  Type     Reason                Age                  From                                                                                          Message
  ----     ------                ----                 ----                                                                                          -------
  Normal   Provisioning          35s (x9 over 4m48s)  csi.vsphere.vmware.com_42<>  External provisioner is provisioning volume for claim <namespace>/<pvc name>"
  Warning  ProvisioningFailed    35s (x9 over 4m48s)  csi.vsphere.vmware.com_<>  failed to provision volume with StorageClass "wcp-policy": rpc error: code = Internal desc = failed to create volume. Error: failed to add task to list view. err: failure to add task to listview. task: Task:task-<task>, err: ServerFaultCode: The session is not authenticated.
  Normal   ExternalProvisioning  2s (x21 over 4m48s)  persistentvolume-controller                                                                   Waiting for a volume to be created either by the external provisioner 'csi.vsphere.vmware.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
• csi controller logs shows same error:
  {"level":"error","time":"YYYY-MM-DD<time>","caller":"volume/listview.go:375","msg":"WaitForUpdates returned err: destroy property filter failed with ServerFaultCode: The session is not authenticated. after failing to wait for updates: ServerFaultCode: The task was canceled by a user. for vc: <VC FQDN>","TraceId":"<Trace-ID>","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/volume.(*ListViewImpl).listenToTaskUpdates\n\t/build/mts/release/bora-24583338/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/volume/listview.go:375"}

Environment

vSphere 7.x

vSphere 8.x

Cause

• After a vCenter reboot  or restart of vsan-health the session is no longer valid.
• This situation may also occur after upgrading/patching ESXI hosts.

Resolution

• CSI controller needs to be restarted in order for a new session to be created.
  kubectl rollout restart deployment -n vmware-system-csi vsphere-csi-controller
• To resolve the ListView issue, upgrade to vCenter Server 8.0 Update 3h or later (including Update 3i).

Additional Information

Details on the issue in https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/3064