Restore points fail to load or experience latency in VMware Cloud Foundation Operations

Products

• VCF Operations

Issue/Introduction

In VMware Cloud Foundation (VCF) Operations, when navigating to the Fleet components Backup and Restore page, past restore points fail to load or appear missing.

The Restore page experiences severe latency, spinning continuously for several minutes before failing or timing out

Environment

VCF Operations 9.1

Cause

This issue occurs due to slow SFTP directory traversal when querying backup manifests:

• NetApp Snapshot Chain Scanning: The backup target volume has client-visible snapshots (~snapshot) enabled. When VMSP queries for backup manifests, it scans the entire snapshot chain, significantly increasing directory walk time.

• File Volume Latency: A large volume of stale or non-backup files within /backup_vcf/ increases listing time to 2.5–4+ minutes per request.

• Polling Timeout Exhaustion: Fleet LCM enforces a strict 5-minute (300-second) polling timeout limit. The extended directory scan times and request queuing cause tasks to exceed 300 seconds, resulting in a 100% failure rate.

Resolution

To resolve this issue, perform the following steps to reduce SFTP directory traversal time:

1. Clean up the SFTP storage directory:

   • Access the remote SFTP server hosting the backup directory.

   • Navigate to the /backup_vcf/ directory.

   • Prune, delete, or archive old, stale, or non-backup files to reduce total directory file count.

2. Hide NetApp snapshot directories:

   • Log in to the NetApp storage management interface or CLI.

   • Configure the backup volume settings to hide or disable client visibility for the ~snapshot directory. This prevents VMSP from traversing snapshot chains during manifest searches.

3. Verify directory performance:

   • Ensure directory listing response times on the SFTP target complete well below the 300-second Fleet LCM timeout limit.

   • If latency persists, temporarily deploy a secondary local SFTP server to verify whether the delay is caused by storage I/O bottlenecks or network latency.

Additional Information

Subscribe to this knowledge article to get updates on this issue