Best practices for manual remediation of stranded control plane nodes - VCF
Article ID: 451796
Updated On:
Products
Issue/Introduction
Failed upgrades in VCF 9.1 environments can cause control plane nodes to become stranded, creating placement-mismatch loops in VCF services runtime logs and Lifecycle Management UI tasks.This issue typically occurs after failed Storage vMotion (SvMotion) operations, which leave virtual machines (VMs) or First Class Disks (FCD) residing in unexpected directories. Manual remediation using the Datastore Browser to delete files within VM home directories must be avoided, as this workaround introduces a high risk of catastrophic data loss.
Environment
- VCF Operations 9.1.0
- VIDB 9.1.0
Cause
Manual Datastore Browser deletions on stranded control plane nodes cause severe cluster-wide I/O errors by accidentally deleting active First Class Disks (FCDs), as these cleanups do not verify directory contents before removal.When FCDs are co-resident in the same home directory as the virtual machine (VM), they become highly vulnerable during these free-form cleanup tasks. While a Storage vMotion (SvMotion) operation successfully updates the FCD catalog record with the correct disk path ID (####-####), the physical VMDK and VMFD files remain targetable and are easily destroyed during unverified manual directory deletions.
Resolution
Prioritize automated, FCD-aware recovery tooling and completely avoid manual file deletions (such as VMDK, VMFD, or flat files) via the vCenter Datastore Browser.Because First Class Disks (FCD) operate independently of traditional folder naming conventions, manual cleanup tasks must be hardened to strictly validate FCD presence before any directory removal occurs to preserve vital persistent storage. If a control plane node becomes stranded, immediately engage Broadcom Support for structured recovery procedures to protect the integrity of the VCF services runtime and prevent cluster-wide I/O failures. To engage Broadcom Support refer to Creating and managing Broadcom cases
Additional Information
Feedback
