FCD catalog resync and data recovery fails after Minio3 pod redeployment

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• After redeploying the minio3 pod, attempts to resync the First Class Disk (FCD) catalog and vClock do not yield new data recovery.
• The expected data remains invisible on the vSAN layer.
• Database observation indicates that the vclock values match the existing database records, showing no new objects detected.

Environment

TKG 2.5.1

Cause

• The system is unable to detect or register new First Class Disk (FCD) objects from the vSAN layer.
• This is due the the datastore vclock values and the existing vCenter Database (VCDB) records are same and the vCenter Storage Lifecycle Manager (VSLM) skips the synchronization process.

 

Resolution

1. Identify all affected datastores and connected ESXi hosts.

2. Open SSH session  to all the ESXi host and stop the host daemon:
   
   /etc/init.d/hostd stop

3. Backup the existing catalog files to a temporary location from any one ESXi host for all the affected datastores.
   
   mkdir /tmp/<datastore_name>-catalog-bkp cd /vmfs/volumes/<datastore_name>/catalog mv $(ls | grep -v journal) /tmp/catalog

4. Restart the hostd service on the all the hosts:
   
   /etc/init.d/hostd start.

5. Enable the Managed Object Browser (MOB) on the ESXi host:
   
   vim-cmd hostsvc/advopt/update Config.HostAgent.plugins.solo.enableMob bool true

6. Navigate to the ESXi Host MOB URL to execute datastore inventory reconciliation:
   
   https://<Host IP address>/mob/?moid=ha-vstorage-object-manager&method=reconcileDatastoreInventory

7. Input the Datastore UUID and run the task until it succeeds for each datastore.

8. Verify that the catalog folder and vclock files are regenerated in the datastore path.
   
   Navigate to cd /vmfs/volumes/<datastore_name>/catalog/vclock

9. Establish an SSH session to the vCenter Server and access the VCDB:
   
   psql -U postgres -d VCDB

10. Reset the vClock values to force vCenter to acknowledge the storage changes:
    
    update cns.vpx_storage_datastore_info set vclock=-1 where datastore_url='<datastore_url>'; delete from cns.volume_info where datastore='<datastore_url>';

11. Restart the vSAN health service:
    
    vmon-cli --restart vsan-health

12. Monitor and verify that the vclock values update and volumes populate in the cns.volume_info table.
    
    The below query displays the 5 most recently updated volumes
    
    SELECT volume_id, volume_name, vclock 
FROM cns.volume_info 
ORDER BY vclock DESC 
LIMIT 5;

13. If the database remains stagnant, manually trigger a full synchronization via the StorageLifecycleManager MOB:
    
    https://<REDACTED_IPS>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore

14. Input the datastore URL, set fullSync=true, and leave the fcd Id field blank.

15. Redeploy the minio3 pod and attempt to synchronize data.