prometheus pod stuck on ContainerCreating due to PV attachment

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pod creation is stuck on ContainerCreating status.
• Describe the pod with below similar messages:
  
  Warning FailedMount 40m kubelet Unable to attach or mount volumes: unmounted volumes=[storage-volume, unattached volumes [storage-volume config-volume kube-api-access-g4mc8]: timed out waiting for the condition
  
  
• Get the log of CSI controller pod which indicates the Volumeattachments stuck on deletion:
  
  I0429 01:45:18.688256 1 csi_handler.go:231] Error processing "csi-<UUID>": failed to detach: rpc error: code = Internal desc = Watch on virtualmachine "<worker-node>" timed out

Environment

vSphere with Tanzu

Resolution

To resolve the issue:

1. Login Supervisor cluster:
   
   kubectl vsphere login --server <Supervisor-VIP> --vsphere-username <SSO_Username> --insecure-skip-tls-verify


2. Get the Volumeattachments which has the problem:
   
   kubectl get volumeattachments.storage.k8s.io


3. Edit the volumeattachments:
   
   kubectl edit volumeattachments.storage.k8s.io csi-<uuid>
   
   
4. Delete the finalizers value.
   
   For example:
   
     finalizers:
  - external-attacher/csi-vsphere-vmware-com      <<< this line need to be deleted
  name: csi-<uuid>
   
   To: 
   
     finalizers:
  name: csi-<uuid>


5. Save the change.
6. Verify the Volumeattachments has been deleted and prometheus pod creation is successful.

NOTE: deletion of finalizers must need to be caution. suggest to contact Broadcom Support. 

Additional Information

How to Remove Finalizers to Delete Kubernetes Objects