A pod in the guest cluster is stuck in ContainerCreating state due to a Block/RWO volume attachment failure.

Products

VMware vSphere ESXi VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

A pod in the guest cluster is stuck in ContainerCreating state.

When describing the pod, we see a volume attachment failure event because the CNS fails to retrieve the datastore that is backing the volume.

Warning FailedAttachVolume 18s attachdetach-controller AttachVolume.Attach failed for volume "pvc-90b1558e-c3bd-461b-9acc-14ccc2d6567d" : rpc error: code = Internal desc = observed Error: "ServerFaultCode: CNS: Failed to retrieve datastore for vol cd07685d-5170-44ed-ae88-dbd66419b27e. (vim.fault.NotFound) {\n faultCause = (vmodl.MethodFault) null, \n faultMessage = <unset>\n msg = \"The vStorageObject (vim.vslm.ID) {\n dynamicType = null,\n dynamicProperty = null,\n id = cd07685d-5170-44ed-ae88-dbd66419b27e\n} was not found\"\n}" is set on the volume "577b7137-41f6-4c35-ba26-48316a458f15-90b1558e-c3bd-461b-9acc-14ccc2d6567d" on virtualmachine "tkgs-cluster-1-worker-nodepool-a1-js6t6-55845c4bb5-c78m5"

The volume metadata is missing from the Pandora database on the vCenter.

root@vcsa1 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT * from vpx_storage_object_info where id='cd07685d-5170-44ed-ae88-dbd66419b27e'"
id | name | capacity | datastore_url | create_time | v_clock | backing_object_id | disk_path | used_capacity
----+------+----------+---------------+-------------+---------+-------------------+-----------+---------------
(0 rows)

Note: Due to the para-virtualized architecture of the CSI in the guest clusters, each volume is represented by two names: one is generated by the pvCSI in the guest cluster and the other is generated by the CSI in the supervisor cluster. The latter is the value that is stored in the database. Therefore, don't filter by the volume name that was mentioned in the pod events. Instead, use the volume ID. If the pod events didn't mention that ID, please check the resolution section for more details on how to get it.

The vmdk that is backing the volume exists in the datastore. To check:

- Take the first part of the volume ID and separate them as ("cd 07 68 5d").
- SSH to one of the ESXi hosts and search for the vmdk file containing the FCD ID.

- If the vmdk exists, it prints the complete path of the VMDK containing the datastore UUID.

/vmfs/volumes/61717a37-7f97f70e-9bce-0050560181cb/fcd/458d9a81daf54a41a10ed09e392d71a1.vmdk
ddb.fcd.uuid = "cd 07 68 5d 51 70 44 ed-ae 88 db d6 64 19 b2 7e"

Note: Get the datastore name. It will be needed as part of the solution.

[root@esxi:~] localcli storage filesystem list | grep -i 61717a37-7f97f70e-9bce-0050560181cb
/vmfs/volumes/61717a37-7f97f70e-9bce-0050560181cb Tanzu 61717a37-7f97f70e-9bce-0050560181cb true VMFS-6 241323474944 5274337280

In this example, the datastore's name is Tanzu.

Environment

VMware vSphere 7.0 with Tanzu

Cause

The volume is present in the backend datastore but missing from Pandora database

Resolution

Upgrade to a version vCenter Server 8.0 Update 3e or above as the Pandora db is no longer used in vSphere 8.0

Workaround:

Identify the volume ID, if it isn't mentioned in the pod events.

Remember: The guest clusters utilize a para-virtualized CSI. This means the PV on the guest clusters refers to a PVC on the supervisor cluster. This PVC will be bound to a PV refers to the volume ID.

- Describe the problematic volume in the guest cluster and get the VolumeHandle.

tseadmin@kube-cli:~$ kubectl describe pv pvc-90b1558e-c3bd-461b-9acc-14ccc2d6567d | grep -i VolumeHandle
VolumeHandle: 577b7137-41f6-4c35-ba26-48316a458f15-90b1558e-c3bd-461b-9acc-14ccc2d6567d

- Go to the supervisor cluster and get the PV that is bound to this PVC.

root@42320f0e4760472d1a96bbbd0bdaa921 [ ~ ]# kubectl get pvc -A | grep -i 577b7137-41f6-4c35-ba26-48316a458f15-90b1558e-c3bd-461b-9acc-14ccc2d6567d
tanzu-1 577b7137-41f6-4c35-ba26-48316a458f15-90b1558e-c3bd-461b-9acc-14ccc2d6567d Bound pvc-1e2b8a72-7b9e-40ae-8ce6-eb61c510913f 1Gi RWO tanzu 50d

- Get the VolumeHandle of this PV, this is our volume ID.

root@42320f0e4760472d1a96bbbd0bdaa921 [ ~ ]# kubectl describe pv pvc-1e2b8a72-7b9e-40ae-8ce6-eb61c510913f | grep -i VolumeHandle
VolumeHandle: cd07685d-5170-44ed-ae88-dbd66419b27e

Note: We can identify the ID from the pvCSI controller logs or the CNS logs (vsanvcmgmtd.log). However, I recommend the above steps because the logs may be rolled over.

Given the datastore name that is backing the volume, get the managed object ID (MOID) of that datastore by running this command on the vCenter.

In this example, the MOID is datastore-2009.

Go to https://<vcip>/mob and log in with the SSO administrator account. Click “content”, then “VStorageObjectManager”, then "VCenterUpdateVStorageObjectMetadataEx_Task”. Insert the volume ID, the datastore MOID and the metadata KeyValue as shown below, then click Invoke Method.

Make sure that the DB has been updated successfully.

root@vcsa1 [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT * from vpx_storage_object_info where id='cd07685d-5170-44ed-ae88-dbd66419b27e'"
id | name | capacity | datastore_url | create_time | v_clock | backing_object_id | disk_path | used_capacity
--------------+-------------------+---------+-----------------------------+----------------+---------+-------------------+---------------------------------------------------+---------------
cd07685d-5170-44ed-ae88-dbd66419b27e | pvc-1e2b8a72-7b9e-40ae-8ce6-eb61c510913f | 1024 | ds:///vmfs/volumes/61717a37-7f97f70e-9bce-0050560181cb/ | 2022-09-13 21:16:54.459 | 91 | | [Tanzu] fcd/458d9a81daf54a41a10ed09e392d71a1.vmdk | -1
(1 row)

Wait 2-3 minutes and check the pod status. If it didn't run, then delete the pod and recreate it.

If the issue persists, then there may be some discrepancies with the disk catalog, refer Reconciling Discrepancies in the Managed Virtual Disk Catalog