VCF upgrade failed during the component deployment phase due to a timeout

Products

• VCF Automation

Issue/Introduction

1.The VCFA upgrade process timed out after 6 hours

2.The health-status-app DaemonSet failed to schedule correctly due to an invalid node selector, resulting in service endpoint loss

3.Because the DaemonSet pods failed to run, the health-status-service lacked valid endpoints, causing the VCF health verifier job to error out

4.Historical failed pods in prelude and vmsp-platform namespaces prevented clean reconciliation of the Helm releases

Environment

VCF Automation 9.1.x

Cause

The health-status-app DaemonSet was misconfigured with an invalid nodeSelector key ("non-existing"), preventing pod deployment and causing health service dependency failures

Resolution

1)  Inspect current DaemonSet configuration:
    kubectl get ds -n prelude health-status-app

2)  Strip the selector permanently since DESIRED is 0 and NODE SELECTOR displays non-existing=true:
    kubectl patch ds health-status-app -n prelude --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'

3)  Verify the DaemonSet scales to match the node count and all pods transition to 1/1 Running:
    kubectl get ds -n prelude health-status-app
    kubectl get pods -n prelude -l app=health-status-app -o wide

4)  Confirm the health-status-service reports active endpoints:
    kubectl get ep -n prelude health-status-service

5)  Purge all historical failed, evicted, or terminated pods across the prelude and vmsp-platform namespaces:
    kubectl delete pods -n prelude --field-selector=status.phase=Failed --force --grace-period=0
    kubectl delete pods -n vmsp-platform --field-selector=status.phase=Failed --force --grace-period=0
    kubectl delete pods -n prelude -l app=vcfa-health-verifier --force --grace-period=0

6)  Force Flux to reconcile vmsp-prelude-deployer and vksm-stack in helm releases:
    kubectl annotate hr vmsp-prelude-deployer -n prelude reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
    kubectl annotate hr vksm-stack -n prelude reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite

7)  Confirm all prelude HelmReleases return READY: True:
    kubectl get hr -n prelude

8)  Verify synthetic health endpoint via loopback:
    curl -k https://127.0.0.1:30006/status | jq .

9) Trigger the precheck upgrade again on the VCFA component from the VCF OPS UI.

10) Trigger the upgrade and monitor the process.