vRealize Automation upgrade fails due to vco-app-xxxx is not in Ready or Completed state

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• vRealize Lifecycle Manager returns the error during the last stages of the health check that upgrade has failed and that you need to revert to snapshot:

Error code: LCMVRAVACONFIG90030
vRealize Automation VA Upgrade Status Check failed
Upgrade on vRealize Automation VA <node> with state fatal

• The vRealize Automation "upgrade-report-latest" log shows output similar to below.
• The upgrade appears to be a success but health check failed because of the vco pods on the secondary. nodes;

 

Overall Status:     Normal

Hostname:        <vRealize Automation_FQDN>

Status:         Upgraded

Cluster Member:     Yes

Version Before:     8.9.0.24128

Version After:     8.9.1.24538

Description:      The node is upgraded successfully.

 

Hostname:        <vRealize Automation_FQDN>

Status:         Upgraded

Cluster Member:     Yes

Version Before:     8.9.0.24128

Version After:     8.9.1.24538

Description:      The node is upgraded successfully.

 

Hostname:        <vRealize Automation_FQDN>

Status:         Upgraded

Cluster Member:     Yes

Version Before:     8.9.0.24128

Version After:     8.9.1.24538

Description:      The node is upgraded successfully.

 

Errors

------------------------------------------------------------------------------------

[ERROR][2023-03-03 10:47:39][<vRealize Automation_FQDN>] Pod: vco-app-xxxxxx is not in Ready or Completed state. All pods must be in either of these states.

[ERROR][2023-03-03 10:47:39][<vRealize Automation_FQDN>] Pod: vco-app-xxxxxx is not in Ready or Completed state. All pods must be in either of these states.

[ERROR][2023-03-03 10:47:39][<vRealize Automation_FQDN>] Services verification found errors.

 

Environment

VMware vRealize Automation 8.x

Cause

This is a known issue when the vco pods are delayed on startup, resulting in a health check failure even though the upgrade is a success.

Resolution

The Issue is resolved in the vRealize Automation 8.10.1 release

Workaround:

To workaround this issue resume the upgrade on the vRealize Automation appliance with command below.

vracli upgrade exec --resume

Once you have ran the command above and the upgrade is a success you can clear the request in vRealize Life Cycle Manager  using swagger API or retry the request with revert to snapshot option set to "false"

Additional Information

Impact/Risks:

vRealize Lifecycle Manager shows the upgrade as failed and prompts a revert to snapshot and no option to skip the process.