VCF management components fail to load in the VCF Operations Lifecycle portal following a cluster reboot

Products

• VCF Operations

Issue/Introduction

- VCF Fleet Management UI is blank

- VMSP nodes failed to uncordon automatically after a planned restart and VMSP nodes report a Ready, SchedulingDisabled status

- Run vcf_services_runtime_shutdown.sh script failed due to "Timed out waiting for task xxx after 600s"

- Manual inspection confirmed the shutdown task was still registered as "Running" despite the script termination.

Environment

VCF Operations 9.1

Resolution

1. Manually delete the stalled system-shutdown task object from the Kubernetes cluster.
2. Execute kubectl uncordon for all affected VMSP nodes refer to KB440862
3. If run cluster-manual-recovery.sh failed due to "power-off-market" not found, refer to docs to shutdown VMSP runtime cluster and power on the VMSP runtime cluster again.
4. If vcf_services_runtime_shutdown.sh failed due to timeout, then run cluster-manual-recovery.sh again because  "power-off-market" will be generated then.
5. In compared with other healthy VMSP runtime cluster to check if any deployment replic missing, and manually scale the found management deployments (e.g., depot-service, vidb-service) to the desired replica count.
6. Verify VMSP Management UI accessibility in VCF Operations.