Error: vcfa-health-verifier pods in Error state

Products

• VCF Automation
• VCF Operations

Issue/Introduction

Possible symptoms:

• When you list system pods, a number of vcfa-health-verifier pods in the prelude namespace show a status of Error.

  kubectl get pods -n prelude

• When you send a GET request to https://$VCFA_FQDN:3006/status for a health check, it shows vcfa-services-prelude-health-reporter-http has "ok": false.
  From the appliance:

  curl -k https://$(kubectl get nodes -ojson | jq -r '.items[].status.addresses[] | select(.type == "ExternalIP").address' | shuf -n 1):30006/status | jq -r

• You may encounter this from Fleet Management with the following error:

  Error Code: LCMVMSP10036
  Applying product patch failed.
  -- PACKAGE DEPLOYMENT [prelude/vcfa-bundle] STATUS: Progressing -- package deployment is in progress -- BUNDLE [prelude/vcfa-bundle] STATUS: Pushed -- -- HR [prelude/vksm-stack] READY: False (DependencyNotReady) -- dependency 'prelude/vmsp-prelude-deployer' is not ready -- HR [prelude/vmsp-prelude-deployer] READY: Unknown (Progressing) -- Running 'install' action with timeout of 1h0m0s

Environment

VMware Cloud Foundation Automation 9.x

Cause

Pods are in a failed status, such as Unknown, Evicted, or Error.

Resolution

The system does not automatically delete or restart pods in a failed status other than Error.

1. Find pods that are in a bad state:

   kubectl get pods -A | grep -v Running | grep -v Completed

2. Delete any pods that are in Unknown or Evicted status:

   kubectl delete pods <pod1> <pod2> -n <namespace>

3. Allow time for the system to restart pods and rerun health checks.
4. If the vcfa-health-verifier pods do not start completing successfully, there may be pods stuck in an Error state. Contact Broadcom Support for help with resolving other VCFA health issues:
   • Creating and managing Broadcom support cases