CNS volume not found in-memory but exists in database

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

Note: Only follow this guide if both the symptoms show up

1. The below log is observed in vsanvcmgmtd.log, where cb96305a-####-####-####-########1d4 is the problematic volume id.

vsanvcmgmtd-251.log.gz:2023-03-22T20:40:21.680Z error vsanvcmgmtd[03552] [vSAN@6876 sub=FcdService opId=04189b21] Failed to get volume storage info for cb96305a-####-####-####-########1d4. Skipping
vsanvcmgmtd-251.log.gz:2023-03-22T20:43:47.706Z error vsanvcmgmtd[06414] [vSAN@6876 sub=FcdService opId=SWI-389be5f7-a512] Failed to get volume storage info for cb96305a-####-####-####-########1d4. Skipping
vsanvcmgmtd-251.log.gz:2023-03-22T20:45:21.758Z error vsanvcmgmtd[11652] [vSAN@6876 sub=FcdService opId=0418aad4] Failed to get volume storage info for cb96305a-352a-####-####-####-########1d4. Skipping
vsanvcmgmtd-251.log.gz:2023-03-22T20:48:47.451Z error vsanvcmgmtd[05957] [vSAN@6876 sub=FcdService opId=SWI-34f62199-b1b5] Failed to get volume storage info for cb96305a-####-####-####-########1d4. Skipping

 

2. Nonempty results are returned by below query in vcdb, where cb96305a-####-####-####-########1d4 is the problematic volume id.

select * from cns.volume_info where volume_id = 'cb96305a-####-####-####-########1d4';
select * from cns.volume_labels where volume_id = 'cb96305a-####-####-####-########d4';

 

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 8.0.2

Cause

In rare cases, CNS in-memory cache could be purged by mistake, yet the database information is correct and up to date.
This particular concern came into existence as of vCenter 8.0; thus, it remains inconsequential to vCenter 7.0 or any earlier releases.

Resolution

The issue is fixed in vCenter Server 8.0 Update 2.

Workaround:

To work around the issue, follow the instructions mentioned below:

• Make sure the upper layer application can tolerate temporary vsan-health downtime during the restart, or the administrator needs to pause the upper layer application manually.
• Restart vsan-health to solve this issue as CNS loads from database upon process restart.