vcfa-health-verifier pods are in Error state on VCF Automation

Products

• VCF Automation

Issue/Introduction

Possible symptoms:

• Listing system pods shows a number of vcfa-health-verifier pods in the prelude namespace which have status: Error
  • kubectl get pods -n prelude
• Sending a GET request to https://$VCFA_FQDN:3006/status for health check shows vcfa-services-prelude-health-reporter-http has "ok": false
  From the appliance:
  • curl -k https://$(kubectl get nodes -ojson | jq -r '.items[].status.addresses[] | select(.type == "ExternalIP").address' | shuf -n 1):30006/status | jq -r
• This may be encountered from Fleet Management with the following error:

  • Error Code: LCMVMSP10036
Applying product patch failed.
-- PACKAGE DEPLOYMENT [prelude/vcfa-bundle] STATUS: Progressing -- package deployment is in progress -- BUNDLE [prelude/vcfa-bundle] STATUS: Pushed -- -- HR [prelude/vksm-stack] READY: False (DependencyNotReady) -- dependency 'prelude/vmsp-prelude-deployer' is not ready -- HR [prelude/vmsp-prelude-deployer] READY: Unknown (Progressing) -- Running 'install' action with timeout of 1h0m0s

Environment

VMware Cloud Foundation Automation 9.x

Cause

There are pods in failed status such as Unknown, Evicted or Error

Resolution

Pods which are in failed status other than Error will not be deleted/restarted automatically by the system.

1. Find pods which are in a bad state:
   • kubectl get pods -A | grep -v Running | grep -v Completed
2. Delete any pods which are in Unknown or Evicted status:
   
   • kubectl delete pods <pod1> <pod2> -n <namespace>
3. Leave the system some time to restart pods and rerun health checks.
4. If the vcfa-health-verifier pods do not start completing successfully, there may be pods which are stuck in Error state.
   Engage Broadcom Support referencing this article (#429231) for help with resolving other VCFA health issues.
   
   • Creating and managing Broadcom support cases