Automatic IaaS upgrade to vRA 7.6 fails on upgrade-server task for primary web node - all IaaS server components are already vRA 7.6

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
The automatic upgrade to 7.6 fails on upgrade-server task for primary IaaS web node and the error does not contain a recommendation to revert the node and the IaaS database.
The "revertRequired" field in the "/opt/vmware/var/log/vami/upgrade-iaas.json" output file is "false" for the command that failed.
The VAMI Cluster tab displays all server components (Database, Website, Model Manager Web, Model Manager Data, Wapi, Manager Service) that reside on the primary node as upgraded to the new version.

Note: This does not include DEM and Proxy agent components, if any reside on the same node, since they are being upgraded on a later stage with a different command and are expected to still be the old version at this point.

Environment

VMware vRealize Automation 7.6.x

Cause

That is not an uncommon situation and is usually caused by connectivity issues between primary IaaS web node and CAFE and/or wrong load balancer settings.

Resolution

This is one of the situations where retry without reverting anything is allowed.
After troubleshooting the failure and making sure the root cause is not present anymore - upgrade could be retried but that must happen manually - since all server components on the primary node are already upgraded and the automatic upgrade will not start an upgrade command for that node.

To retry the primary web upgrade manually execute the following steps:

1. SSH to primary VA.
2. Execute "vra-command list-nodes" (or its verbose versions "vra-command list-nodes --components") to get the node ID of the primary web node.
3. Execute "vra-command execute --node <primary_web_node_id> upgrade-server" for upgrades from 7.x

or,

Execute "vra-command execute --node <primary_web_node_id> upgrade-server --VraAddress"<vRA_LB_hostname>" --VraWebCertificateThumbprint "<vRA_certificate_thumbprint>" --DefaultTenant "<vRA_default_tenant>" --VidmAdminUser "administrator" --VidmAdminPassword "<vIDM_admin_password>"" for upgrades from 6.2.x where:

vRA_LB_hostname - is the hostname of the VA load balancer (or the VA hostname if not in HA) - this can be retrieved by executing "/usr/sbin/vcac-vami host info" command on the primary VA

 

vRA_certificate_thumbprint - is the thumbprint of the vRA certificate without spaces and dashes and with capital letters

 

vRA_default_tenant - is the default vRA tenant and can be retrieved from the following configuration file on the primary VA "/etc/vcac/security.properties" - set under "vmidentity.websso.tenant" key

 

vIDM_admin_password - is the password for the vIDM administrator

The manual server upgrade will continue from the last successful post upgrade task. If the command fails on the same post-upgrade task as the last time or moves past it at fails on a subsequent one (this could be monitored in the Management Agent All.log file on the primary web node) - it could be retried manually the same way.
 

Note: In case retrying does not resolve the problem and the failure is consistent - revert of the primary web node and the IaaS MS SQL Server database might still be necessary.

4.  After the primary web server upgrade is completed manually - the automatic IaaS upgrade can be re-triggered by executing "/usr/lib/vcac/tools/upgrade/upgrade" script on the primary VA and it will skip the already upgraded node(s).