After the vCenter upgrade from vCenter 7.0U3 to 8.0 U3 unable to create POD with volume attachment - stuck in ContainerCreating State with error "The input volume xxxxxxxx is not registered as a CNS volume".
Article ID: 383364
Updated On:
Products
Issue/Introduction
kubectl describe pod <pod name> -n <namespace>AttachVolume.Attach failed for volume "pvc-xxxxxxxxxxxxxxxxxxxxx" : rpc error: code = Internal desc = failed to attach disk: "xxxxxxxxxxxxxxxxxxxxxxxxx" with .node: "xxxxxxxxxxxxxxxxxxx" err ServerFaultCode: The object or item referred to could not be found.
After restarting POD:
kubectl describe pod <pod name> -n <namespace>
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedAttachVolume 58m attachdetach-controller AttachVolume.Attach failed for volume "pvc-xxxxxxxxxxxxxxx" : rpc error: code = Internal desc = failed to attach disk: "xxxxxxxxxxxxxxxxx" with node: "xxxxxxxxxxxxxxxxxxx" err failed to attach cns volume: "xxxxxxxxxxxxxxxxxxxxx" to node vm: "VirtualMachine:vm-63600 [VirtualCenterHost: xxxxxxxx, UUID: xxxxxxxxxxxxxxxxxxxx, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-2, VirtualCenterHost: xxxxxxxxxxxxxxxxx]]". fault: "(*types.LocalizedMethodFault)(0xc0004e38a0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.BaseMethodFault) <nil>,\n LocalizedMessage: (string) (len=188) \"Volume with ID xxxxxxxxxxxx is not registered as CNS Volume. Error message: The input volume xxxxxxxxxxxxxxxxxx is not registered as a CNS volume.\"\n})\n". opId: "330d28cf"
The following error will be noticed in the vsanvcmgmtd.log (/var/log/vmware/vsan-health/vsanvcmgmtd.log) on the vCenter Server:
"DB Error while inserting volume ***: Column ALLOCATED_SPACE does not exist in Table Definition of table volume_info" or "Cannot process vStorage objectxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx on ds:///vmfs/volumes/****".
The following logging may also be observed in (/var/log/vmware/vsan-health/vsanvcmgmtd.log) on the vCenter Server:
2025-02-27T14:00:00.635+01:00 error vsanvcmgmtd[xxxx] [vSAN@xxx sub=Workflow opId=xxxxxx] Workflow current action has fault (vslm.fault.VslmFault) {--> faultCause = (vmodl.MethodFault) null,--> faultMessage = <unset>,--> msg = "DB Update Failed"
Login to vPostgres Database on vCenter via SSH client and confirm that column volume_info | allocated_space | -5 | 8 | 0 is missing
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
VCDB=# select * from cns.VPX_TABLE; table_name | col_name | col_type | col_len | pk_flg----------------------------+--------------------+----------+---------+-------- VPX_STORAGE_DATASTORE_INFO | vclock | -5 | 8 | 0 VPX_STORAGE_DATASTORE_INFO | datastore_url | 12 | 255 | 1 volume_info | used_capacity | -5 | 8 | 0 volume_info | mark_for_delete | -7 | 1 | 0 volume_info | volume_type | 4 | 4 | 0 volume_info | volume_name | -9 | 510 | 0 volume_info | static_provision | -7 | 1 | 0 volume_info | volume_id | -9 | 128 | 1 volume_info | datastore | -9 | 510 | 0 volume_info | vclock | -5 | 8 | 0 volume_info | is_cns_volume | -7 | 1 | 0 volume_info | disk_path | -10 | 8190 | 0 volume_info | create_time | 93 | 16 | 0 volume_info | last_modified_date | 93 | 16 | 0 volume_info | backing_object_id | -9 | 120 | 0 volume_info | capacity | -5 | 8 | 0 volume_labels | key | -10 | 640 | 0 volume_labels | volume_id | -9 | 128 | 0 volume_labels | cluster_id | -9 | 128 | 0 volume_labels | value | -9 | 510 | 0 volume_labels | namespace | -10 | 512 | 0 volume_labels | label_type | 4 | 4 | 0 volume_labels | entity_ref | -10 | 512 | 0(23 rows)
Environment
vCenter 8.0.3 build-24262322
ESXi 7.0 U3
Cause
The issue occurs due to an upgrade error when modifying database tables to align with the vCenter 8.x database schema.
Resolution
This issue is fixed in VC 8.0U3 U3e
see VMware vCenter Server 8.0 Update 3e Release Notes
Workaround:
Take an offline snapshot of vCenter appliance (or all vCenters in ELM environment) before performing database changes.
Truncate the cns.VPX_TABLE to refill it with the proper information and restart of the vsan health service following the below detailed steps.
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgresTRUNCATE TABLE cns.VPX_TABLE;2. Exit database and restart vsan-health service on vCenter:
vmon-cli -r vsan-healthdatastoreUrl will be ds:///vmfs/volumes/xxxxxxxxxxxxxx/, fullSync should be true and fcdId should be blank. Click "Invoke Method".
4. Login again to postgres database in vCenter and confirm that cns.VPX_TABLE has now the allocated_space column:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgresSELECT * from cns.VPX_TABLE;VCDB=# select * from cns.VPX_TABLE; table_name | col_name | col_type | col_len | pk_flg-----------------------------+--------------------+----------+---------+-------- volume_info | allocated_space | -5 | 8 | 0 volume_info | create_time | 93 | 16 | 0 volume_info | vclock | -5 | 8 | 0 volume_info | capacity | -5 | 8 | 0 volume_info | volume_type | 4 | 4 | 0 volume_info | volume_name | 12 | 255 | 0 volume_info | mark_for_delete | -7 | 1 | 0 volume_info | static_provision | -7 | 1 | 0 volume_info | datastore | 12 | 255 | 0 volume_info | is_cns_volume | -7 | 1 | 0 volume_info | disk_path | -1 | 8190 | 0 volume_info | volume_id | 12 | 255 | 1 volume_info | used_capacity | -5 | 8 | 0 volume_info | backing_object_id | 12 | 240 | 0 volume_info | last_modified_date | 93 | 16 | 0 volume_labels | key | -1 | 1280 | 0 volume_labels | id | -5 | 8 | 1 volume_labels | entity_ref | -1 | 1024 | 0 volume_labels | namespace | -1 | 1024 | 0 volume_labels | cluster_id | 12 | 255 | 0 volume_labels | value | 12 | 255 | 0 volume_labels | label_type | 4 | 4 | 0 volume_labels | volume_id | 12 | 255 | 0 VPX_STORAGE_VOLUME_METADATA | value | -1 | 16384 | 0 VPX_STORAGE_VOLUME_METADATA | key | -1 | 1024 | 1 VPX_STORAGE_VOLUME_METADATA | volume_id | 12 | 255 | 1 VPX_STORAGE_DATASTORE_INFO | vclock | -5 | 8 | 0 VPX_STORAGE_DATASTORE_INFO | datastore_url | 12 | 255 | 1(28 rows)5. Wait for 10 minutes and check if volumes are listed in in cns.volume_info table:
select * from cns.volume_info;Feedback
