Snapshotted Cloud Native Storage volume disappears from UI after attempting delete

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Typically in /var/log/vmware/vsan-health/vsanvcmgmtd.log the below log is found:

2023-02-08T21:49:12.482Z error vsanvcmgmtd[44176] [vSAN@6876 sub=Workflow opId=462ac6db] Workflow current action has fault (vim.fault.CnsFault) {
  faultCause = (vmodl.MethodFault) null,
  faultMessage = <unset>,
  reason = "Failed to delete volume. Error: (vmodl.fault.InvalidArgument) {
  faultCause = (vmodl.MethodFault) null,
  faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
     key = "com.vmware.vim.fcd.error.snapshotsNotAllowed",
     arg = (vmodl.KeyAnyValue) [
       (vmodl.KeyAnyValue) {
        key = "snapshot",
        value = "3d 9f 1d 5a 71 52 43 c1-b0 f7 7f d7 79 fc 2c 4e"
       }
     ],
     message = "Cannot be performed on FCD with snapshots. Snapshot 3d 9f 1d 5a 71 52 43 c1-b0 f7 7f d7 79 fc 2c 4e relies on this FCD"
    }
  ],
  invalidProperty = "id"
  msg = "A specified parameter was not correct: id"
 }"
  msg = ""
 }

And the volume disappears from CNS UI and cns query api.

• Verify the DB content to observe the field is marked as True with the below SQL

select mark_for_delete from cns.volume_info where volume_id ='missing_pv_id';

Environment

VMware vCenter Server 7.0.x

Cause

Due to an internal sync logic, a volume gets marked as pending delete before the delete call to lower layers return successful. When there is snapshot on this volume, lower layers throw exception that is not handled successfully. The volume gets stuck in pending delete state.

Resolution

The issue has been resolved in vCenter server 8.0 and higher versions.

Workaround:

To workaround the issue, run the below sql command after logging in to vCenter DB:

update cns.volume_info set mark_for_delete=false where volume_id='missing_pv_id';

This will bring back the volume from pending delete state and appear in CNS UI and query api.