Application Pods Fail to Initialize Due to PVC Attachment Timeout

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Application pods are failing to initialize and remain in an Init:0/2 state due to Persistent Volume Claim (PVC) attachment failures.
  
  
• Describing the pod in Init state gives the following error:
  
  Warning  FailedAttachVolume    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-####" : rpc error: code = Internal desc = observed Error: "time out for task Task:task-<id> before response from CNS" is set on the volume "<volume_id>" on virtualmachine "<node_name>"


• Volumeattachment object is in attached "false" state for the PVC.

Environment

vSphere Kubernetes Service

Cause

A synchronization issue between datastore and database.

Resolution

Resync the datastore in the vCenter database.

1. Connect to VCDB
   psql -d VCDB -U postgres

2. Confirm that the volume ID.
   select * from cns.volume_info where volume_name='pvc-<id>';

3. Resync the datastore, using MOB
   https://<VCIP>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore
   
   

4. Datastore url is taken from VC UI and looks like ds:///vmfs/volumes/########-########-####-############.
    fullSync=true
 fcdId should be blank

5. Restart vsan-health service
   service-control --restart vsan-health