After the vCenter upgrade from vCenter 7.0U3 to 8.0 U3, unable to create PODs, it is stuck in ContainerCreating State with error "The input volume xxxxxxxx is not registered as a CNS volume".

Products

• VMware vCenter Server

Issue/Introduction

After the vCenter upgrade from vCenter 7.0U3 to 8.0 U3, unable to create PODs, it is stuck in ContainerCreating State with the error "The input volume xxxxxxxx is not registered as a CNS volume".

POD failure events: 

AttachVolume.Attach failed for volume "pvc-xxxxxxxxxxxxxxxxxxxxx" : rpc error: code = Internal desc = failed to attach disk: "xxxxxxxxxxxxxxxxxxxxxxxxx" with .node: "xxxxxxxxxxxxxxxxxxx" err ServerFaultCode: The object or item referred to could not be found.

PV and PVC creation succeeds, volume attachments show Bound but failure in Sync.

The pvc or the volume ID are not available on the VCDB

Restarting pod:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 58m attachdetach-controller AttachVolume.Attach failed for volume "pvc-xxxxxxxxxxxxxxx" : rpc error: code = Internal desc = failed to attach disk: "xxxxxxxxxxxxxxxxx" with node: "xxxxxxxxxxxxxxxxxxx" err failed to attach cns volume: "xxxxxxxxxxxxxxxxxxxxx" to node vm: "VirtualMachine:vm-63600 [VirtualCenterHost: xxxxxxxx, UUID: xxxxxxxxxxxxxxxxxxxx, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-2, VirtualCenterHost: xxxxxxxxxxxxxxxxx]]". fault: "(*types.LocalizedMethodFault)(0xc0004e38a0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.BaseMethodFault) <nil>,\n LocalizedMessage: (string) (len=188) \"Volume with ID xxxxxxxxxxxx is not registered as CNS Volume. Error message: The input volume xxxxxxxxxxxxxxxxxx is not registered as a CNS volume.\"\n})\n". opId: "330d28cf"

Environment

vCenter 8.0.3 build-24262322
ESXi 7.0 U3

Cause

Column allocated_space is missing from cns.VPX_TABLE after vCenter upgrade from 7.0 U to 8.0 U3.

2024-10-16T00:25:06.445+07:00 info vsanvcmgmtd[46656] [vSAN@6876 sub=CnsSync] Sync ds:///vmfs/volumes/xxxxxxxxxxxxxxx/: startVClock = 10506, fullSync = true
2024-10-16T00:25:06.552+07:00 info vsanvcmgmtd[46656] [vSAN@6876 sub=CnsSync] Volume xxxxxxxxxxxxxx at vclock -1 is missing in a full sync result at vclock 1, deleting this stale entry
2024-10-16T00:25:06.553+07:00 info vsanvcmgmtd[46656] [vSAN@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded cns schema and index information from DB
2024-10-16T00:25:06.553+07:00 error vsanvcmgmtd[46656] [vSAN@6876 sub=CnsSync] DB Error while deleting volume xxxxxxxxxxxxxx: Column ALLOCATED_SPACE does not exist in Table Defintion of table volume_info
2024-10-16T00:25:06.553+07:00 info vsanvcmgmtd[46656] [vSAN@6876 sub=CnsSync] Volume xxxxxxxxxxxx at vclock -1 is missing in a full sync result at vclock 1, deleting this stale entry
2024-10-16T00:25:06.562+07:00 info vsanvcmgmtd[46656] [vSAN@6876 sub=CnsSync] Record out-of-band delete/vMotion xxxxxxxxxxxxxxxxx in DB

Resolution

Take an offline snapshot of vCenter appliance (or all vCenters in ELM environment) before performing database changes.

Truncate the cns.VPX_TABLE to refill it with the proper information and restart of the vsan health service following the below detailed steps.

1. Login to postgres in vcenter and Truncate cns.VPX_TABLE.

TRUNCATE TABLE cns.VPX_TABLE;

2. Restart vsan-health.

vmon-cli -r vsan-health

3. Confirm cns.VPX_TABLE has the allocated_space row.

SELECT * from cns.VPX_TABLE;

 

The row we are interested in is:

volume_info | allocated_space | -5 | 8 | 0

4. Run fullSync on the datastore ds:///vmfs/volumes/xxxxxxxxxxxx/ to bring back the missing volumes in cns DB.

https://<vCenter_IP/FQDN>/vslm/mob/?moid=StorageLifecycleManager&method=VslmSyncDatastore

datastoreUrl will be ds:///vmfs/volumes/xxxxxxxxxxxxxx/, fullSync should be true and fcdId should be blank. Click "Invoke Method".

 

5. Wait for 10 minutes and check if volumes in datastore ds:///vmfs/volumes/xxxxxxxxxxxxxxxx/ start showing in cns.volume_info table.