Error Code: LCMVRAVACONFIG90030, vracli error: "Services verification found errors." when upgrading or applying Aria Automation patch

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

If all of the following conditions are met, then the upgrade is complete but the services failed to come up in the end:

• Error Code: LCMVRAVACONFIG90030 when upgrading or applying Aria Automation patch
• Running command 'vracli upgrade status --details' or checking /var/log/vmware/prelude/upgrade-report-latest on the Aria Automation VA shows the following error text:
  • Result:                 Upgrade Failure
Description:            Upgrade failed and left the system in non-working state. Check the error report below to correct the problem. Once addressed, you can continue the upgrade by running 'vracli upgrade exec --resume'
  • In the Cluster section, the upgrade completed successfully on all nodes:
    • Description:            The node is upgraded successfully.
  • However in the Errors section, some pod(s) did not come up successfully:
    • [ERROR][####-##-## ##:##:##][vra.example.com] Pod: ############# is not in Ready or Completed state. All pods must be in either of these states.
[ERROR][####-##-## ##:##:##][vra.example.com] Services verification found errors.
• As an additional check, make sure the 3 Automation nodes are joined as a cluster by listing the kubernetes nodes:
  • kubectl get nodes
• Each node should return the upgraded version number:
  • vracli version

For another possible cause of error LCMVRAVACONFIG90030, see KB article #372059.

Environment

VMware Aria Automation 8.x

Cause

The final stage of the upgrade is to start up the system services. If this fails for any reason, Aria Lifecycle will report a failure, but the upgrade is in fact complete.

Resolution

1. First it is necessary to resolve any issues with the identified pod(s)
   • kubectl -n prelude get pods
   • For assistance with getting these pods back into a working state, contact Broadcom support referencing this article (KB #424939)
     • Creating and managing Broadcom support cases
2. Once addressed, you can continue the upgrade by clicking RETRY in LCM or running the command 'vracli upgrade exec --resume'