Unable to to Take PVC Backup after upgrading CSE from 4.1 to 4.2.3 version.

Products

• VMware Cloud Director

Issue/Introduction

• After upgrading CSE server to version 4.2.3 and the Kubernetes components in CSE clusters using the document: Upgrade Kubernetes Components in VMware Cloud Director Container Service Extension Clusters , PVC backup operations from DELL PPDM fail.

• Restarting the pods in the CSE cluster resolves the issue for that particular cluster.

• Below are the errors observed in CSI driver logs:

  csi_handler.go:620] Saved detach error to "csi-#################################"
  csi_handler.go:228] Error processing "...": failed to detach: persistentvolume "pvc-######################" not found
  csi_handler.go:620] Saved detach error to "csi-###############################"
  csi_handler.go:228] Error processing "...": failed to detach: persistentvolume "pvc-###########################" not found
  controller.go:165] Ignoring VolumeAttachment "... " change
  

 

Environment

Container Service Extension 4.2.3

Cause

After upgrading CSE from 4.1 to 4.2.3 and the Kubernetes components, the CSI driver was upgraded from v1.4 to v1.6. This version change causes PVC backup failures due to issues with volume detach/attach handling in CSI v1.6.

 

Resolution

This is a potential bug in CSI v1.6. VMware engineering teams are investigating to provide a permanent fix in a future CSI release.

As a workaround, downgrade the CSI version to 1.4.

Please raise a Support ticket with Broadcom and refer to this KB article.