CNS throwing "not registered as CNS volume" error when calling updateVStorageObjectMetadata

Products

• VMware vCenter Server

Issue/Introduction

CNS is returning the error "not registered as a CNS volume" for volumes that were not synchronised properly between the CNS DB and the CNS cache on creation. 

Environment

vCenter 7.x
vCenter 8.0

Cause

The 

1. Volume is created and added to CNS DB.

2. Periodic Sync syncs the volume and does not find the volume in the datastore. So it removes the volume from CNS in-memory cache but does not remove it from CNS DB. It is likely there was a vSAN partition which caused the volume to be NotFound from certain hosts the datastore is mounted on. 

vsanvcmgmtd.log:2024-12-31T17:24:34.868+01:00 error vsanvcmgmtd[1486195] [vSAN@6876 sub=FcdService opID=ffbc2829] Could not find disk for fcd id FCD_ID: N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound

vsanvcmgmtd.log:2024-12-31T17:24:34.873+01:00 info vsanvcmgmtd[1486195] [vSAN@6876 sub=Workflow opID=ffbc2829] RetrieveVStorageObjectInfo could not find volume FCD_ID

vsanvcmgmtd.log:2024-12-31T17:24:34.873+01:00 info vsanvcmgmtd[1486195] [vSAN@6876 sub=CnsVolCache opID=ffbc2829] Removed FCD_ID from volume info cache

3. CNS periodically retries syncing the volume. During the retry, the volume is found in the datastore (the vSAN partition would have recovered). Retry sync adds the volume (just the volume id and datastore url) to the CNS cache but fails to insert the volume into CNS DB as it is already present in the DB. Since the DB insert failed, it does not update the cache with the rest of the volume's information such as vclock or is_cns_volume flag. So, the cache has incorrect vclock value of -1 and is_cns_volume flag set to false.

vsanvcmgmtd.log:2024-12-31T17:26:27.548+01:00 info vsanvcmgmtd[1483320] [vSAN@6876 sub=FcdService opId=ffbc657d] RetrieveVStorageObjectInfo disk begin for volumeId: FCD_ID for datastore vim.Datastore:datastore-XXX

vsanvcmgmtd.log:--> DETAIL: Key (volume_id)=(FCD_ID) already exists.;

vsanvcmgmtd.log:2024-12-31T17:26:27.666+01:00 error vsanvcmgmtd[1483320] [vSAN@6876 sub=Default opId=ffbc657d] [VdbStatement] value = "FCD_ID"

vsanvcmgmtd.log:--> DETAIL: Key (volume_id)=(FCD_ID) already exists.;

vsanvcmgmtd.log:2024-12-31T17:26:27.667+01:00 error vsanvcmgmtd[1483320] [vSAN@6876 sub=Workflow opId=ffbc657d] DB Error while inserting volume FCD_ID: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_cns_volume_info"

vsanvcmgmtd.log:--> DETAIL: Key (volume_id)=(FCD_ID) already exists.;

vsanvcmgmtd.log:2024-12-31T17:26:27.667+01:00 info vsanvcmgmtd[1483320] [vSAN@6876 sub=Workflow opId=ffbc657d] Update volume FCD_ID: added = false, updated = false, isCnsVolume = true, vclock = 0, cached vclock = -1

4. UpdateVStorageObjectMetadata or any CNS workflow looks up the volume from the cache and finds that the volume is present with is_cns_volume flag to false. This means the volume is not registered with CNS and hence the error.

Resolution

The resolution will be released in vSphere 8.0u3 P05.

Prior to this, the workaround is to restart the "vsan-health" service on the vCenter.