Persistent Volume connected to Red Hat OpenShift cannot be expanded. Task fails with error : "cannot be expanded. Please delete snapshots before expanding the volume"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Symptoms

Increasing the size of Persistent Volume connected to Red Hat OpneShift fails with the following error: 

volume: ########-####-####-a834-##########28 with existing snapshots [size_bytes:53687091200 snapshot_id:"########-####-####-a834-##########28+########-####-####-ab52-##########2f" source_volume_id:"########-####-####-a834-##########28" creation_time:<seconds:1717251963 nanos:384999000 > ready_to_use:true  size_bytes:53687091200 snapshot_id:"########-####-####-a834-##########28+########-####-####-b4f8-##########30" source_volume_id:"########-####-####-a834-##########28" creation_time:<seconds:1717337094 nanos:899999000 > ready_to_use:true  size_bytes:53687091200 snapshot_id:"########-####-####-a834-##########28+########-####-####-bc91-##########25" source_volume_id:"########-####-####-a834-##########28" creation_time:<seconds:1717509560 nanos:32000000 > ready_to_use:true ] cannot be expanded. Please delete snapshots before expanding the volume

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
vSphere ESXi 7.x
vSphere ESXi 8.x

Cause

• There are no Snapshot visible on the Red Hat OpenShift Operator Console

• Validate if the stale snapshot for the Persistent volume ########-####-####-a834-##########28 are present in the VCDB
  
  
• Log into the VCSA as root via SSH or Console.
  
  
• Type shell to switch to the BASH Shell.
  
  
• Connect to the VCDB by running this command from the vCenter Server Appliance 6.5/6.7/7.0 BASH Shell:
  
  

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

VCDB=# select *  from cns.vpx_storage_snapshot_info where volume_id = '########-####-####-a834-##########28';

-[ RECORD 22 ]----+----------------------------------------------------------------------------------
snapshot_id       | ########-####-####-bc91-##########25
volume_id         | ########-####-####-a834-##########28
disk_path         |
backing_object_id |
create_time       | 2024-06-04 13:59:20.032
description       | snapshot-########-####-####-8a4b-##########a8-########-####-####-a834-##########28
------------------------+-----------------------------------------------------------------------------

-[ RECORD 24 ]----+-----------------------------------------------------------------------------------
snapshot_id       | ########-####-####-b4f8-##########30
volume_id         | ########-####-####-a834-##########28
disk_path         |
backing_object_id |
create_time       | 2024-06-02 14:04:54.9
description       | snapshot-########-####-####-9be8-##########9a-########-####-####-a834-##########28
------------------------+-----------------------------------------------------------------------------

-[ RECORD 25 ]----+-----------------------------------------------------------------------------------
snapshot_id       | ########-####-####-ab52-##########2f
volume_id         | ########-####-####-a834-##########28
disk_path         |
backing_object_id |
create_time       | 2024-06-01 14:26:03.385
description       | snapshot-########-####-####-810d-##########7e-########-####-####-a834-##########28
------------------------+------------------------------------------------------------------------------

• If the Stale snapshot entries are there then they need to be removed from VCDB post confirming with Red Hat Support or with Customer. 

Resolution

• Validate with Customer and Red Hat Support that the Snapshots are not visible on the Red Hat OpenShift Console.
  
  
• Delete the snapshot entries pertaining to Persistent Volume ID. ########-####-####-a834-##########28

VCDB=# delete from cns.vpx_storage_snapshot_info where snapshot_id = '########-####-####-ab52-##########2f';

• Validate the deletion of the entries 

VCDB=# select *  from cns.vpx_storage_snapshot_info where volume_id = '########-####-####-a834-##########28';