Accessing the VAMI returns error "Update installation in progress" after recovering from a failed vCenter update or vCenter version upgrade

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

After logging in to the VAMI UI after recovering from a failed update, instead of the normal Interface, "Update installation in progress" is shown:

the log of application management service (applmgmt.log) has the following entries:

YYYY-MM-DDTHH:MM:SS.255 [10787]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create 
YYYY-MM-DDTHH:MM:SS.259 [10787]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block [] 
YYYY-MM-DDTHH:MM:SS.259 [10787]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation 
YYYY-MM-DDTHH:MM:SS.260 [10787]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
YYYY-MM-DDTHH:MM:SS.263 [10787]INFO:twisted:"127.0.0.1" - - [DD/MM/YYYY:HH:MM:S +0000] "POST /rest/com/vmware/cis/session HTTP/1.0" 200 60 "https://<FQDN>:5480/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0" 
YYYY-MM-DDTHH:MM:SS.267 [10787]INFO:vmware.appliance.vapi.auth:Reset pam tally for root 
YYYY-MM-DDTHH:MM:SS.267 [10787]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login           Failures Latest failure     From\nroot                0    \n' 
YYYY-MM-DDTHH:MM:SS.277 [10787]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.local_accounts, operation_id: get 
YYYY-MM-DDTHH:MM:SS.277 [10787]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ViewConfiguration'] 
YYYY-MM-DDTHH:MM:SS.277 [10787]DEBUG:root:Validated user privileges in localstore or SSO 
YYYY-MM-DDTHH:MM:SS.277 [10787]DEBUG:vmware.vherd.transport.vapi:Only [] block [] 
YYYY-MM-DDTHH:MM:SS.277 [10787]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation 
YYYY-MM-DDTHH:MM:SS.278 [10787]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS' 
YYYY-MM-DDTHH:MM:SS.279 [10787]INFO:twisted:"127.0.0.1" - - [DD/MM/YYYY:HH:MM:SS +0000] "GET /rest/appliance/local-accounts/root HTTP/1.0" 400 136 "https://<FQDN>:5480/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"

The VAMI log (vami.log) contains the following messages:

<response><locale>en-US</locale><requestid>getUpgradeStatus</requestid><value actions="enabled" id="status">firstboot</value><value actions="enabled" id="pr
ogress">100</value><value actions="enabled" id="details">{
    "error": null,
    "info": [],
    "warning": [],
    "start_time": "YYYY-MM-DDTHH:MM:SS.508Z",
    "progress": 100,
    "end_time": "YYYY-MM-DDTHH:MM:SS.369Z",
    "progress_message": {
        "args": [
            "VMware Platform Services Controller Health Monitor"
        ],
        "id": "install.ciscommon.component.starting",
        "localized": "Starting VMware Platform Services Controller Health Monitor...",
        "translatable": "Starting %(0)s...

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x

Cause

Patch Updates: Post patch failure, the appliance has recovered from failed update, and the update state is "INSTALL_IN_PROGRESS" in "/etc/applmgmt/appliance/software_update_state.conf", as in the example below:

{
    "operation_id": "/storage/core/software-update/stage_install_operation",
    "latest_query_time": "YYYY-MM-DDTHH:MM:SSZ",
    "state": "INSTALL_IN_PROGRESS",
    "version": "6.7.0.21000"
}

vCenter Major Version Upgrades: The "INSTALL_IN_PROGRESS" is not shown in the "/etc/applmgmt/appliance/software_update_state.conf" file.

Resolution

Note: This is applicable only for the source vCenter which is restored/powered on after an upgrade failure. Do not apply these steps to recover/workaround the upgrade failure on the failed target vCenter

In order to resolve the issue perform the following:

Login to the appliance using ssh
Create a backup copy of the file /etc/applmgmt/appliance/software_update_state.conf in a different location, e.g. in /storage/core:
```
# cp /etc/applmgmt/appliance/software_update_state.conf /storage/core/software_update_state.conf
```
Stop the VAMI service by running the command:
```
# service-control --stop applmgmt
```
For patch updates, remove the file /etc/applmgmt/appliance/software_update_state.conf :
```
# rm -i /etc/applmgmt/appliance/software_update_state.conf
```
change into the folder /var/log/vmware/upgrade :
```
# cd /var/log/vmware/upgrade
```
verify that the following 3 files exist in this folder:
```
prechecks-succeeded
export-succeeded
import-succeeded
```
If these files do not exist for patch updates or for major version upgrades, run the following 3 commands to create them:
```
# touch prechecks-succeeded
# touch export-succeeded
# touch import-succeeded
```

confirm that the files have been created:

# ls -l   

-rw-r--r-- 1 root root    0 Mar 11  2019 prechecks-succeeded
-rw-r--r-- 1 root root    0 Mar 11  2019 import-succeeded
-rw-r--r-- 1 root root    0 Mar 11  2019 export-succeeded

Start the VAMI service using the following command:
```
 # service-control --start applmgmt
```
Login to the VAMI to validate normal VAMI logon+access has been restored.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ