Accessing the VAMI returns error: "Update installation in progress" after recovering from a failed vCenter update or version upgrade

Products

• VMware vCenter Server

Issue/Introduction

• Logging in to the VAMI after recovering from a failed update or upgrade may show the error "Update installation in progress":
  
  

• The appliance management service logs at /var/log/vmware/applmgmt/applmgmt.log may show "INSTALL_IN_PROGRESS " :

[YYYY-MM-DDTHH:MM:SS] [10787]DEBUG:vmware.vherd.transport.vapi:Only ['INSTALL_IN_PROGRESS'] block [] 
[YYYY-MM-DDTHH:MM:SS] [10787]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation 
[YYYY-MM-DDTHH:MM:SS] [10787]DEBUG:vmware.vherd.transport.vapi:State 'INSTALL_IN_PROGRESS'
[YYYY-MM-DDTHH:MM:SS] [10787]DEBUG:vmware.vherd.transport.vapi:Only [] block [] 
[YYYY-MM-DDTHH:MM:SS] [10787]INFO:vmware.appliance.update.update_state:Found operation in progress /storage/core/software-update/stage_install_operation 

• vami.log may contain the following: 

<response><locale>en-US</locale><requestid>getUpgradeStatus</requestid><value actions="enabled" id="status">firstboot</value><value actions="enabled" id="pr
ogress">100</value><value actions="enabled" id="details">{
    "error": null,
    "info": [],
    "warning": [],
    "start_time": "YYYY-MM-DDTHH:MM:SS.508Z",
    "progress": 100,
    "end_time": "YYYY-MM-DDTHH:MM:SS.369Z",
    "progress_message": {
        "args": [
            "VMware Platform Services Controller Health Monitor"
        ],
        "id": "install.ciscommon.component.starting",
        "localized": "Starting VMware Platform Services Controller Health Monitor...",
        "translatable": "Starting %(0)s...

• In a vSAN enabled cluster, the options under Cluster->Monitor and Cluster->Configuration do not appear in the vSphere UI

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

The issue may occur after a patch upgrade failure.

Post patch failure, the appliance may have recovered from a failed update, and the update state is "INSTALL_IN_PROGRESS" in "/etc/applmgmt/appliance/software_update_state.conf":

 {  
 "state": "INSTALL_IN_PROGRESS",
  "version": "8.0.3.00400",
  "latest_query_time": "YYYY-MM-DDTHH:MM:SSZ",,
  "operation_id": "/storage/core/software-update/install_operation"
}

 

vCenter Major Version Upgrades:  The "INSTALL_IN_PROGRESS" is not shown in "/etc/applmgmt/appliance/software_update_state.conf"

Resolution

Note: This procedure is applicable only to the source vCenter Server that has been restored or powered on following an upgrade failure. Do not use these steps to recover or work around an upgrade failure on the target vCenter Server.

It is strongly recommended to take a backup of the vCenter Server before making any configuration changes.

To resolve the issue:

1. SSH into the vCenter appliance 
   
2. Create a backup copy of /etc/applmgmt/appliance/software_update_state.conf:
   # cp /etc/applmgmt/appliance/software_update_state.conf /storage/core/software_update_state.conf.bak
   
   
3. Stop the appliance management service:
   # service-control --stop applmgmt
   
   
4. Remove the software_update_state.conf:
   # rm -i /etc/applmgmt/appliance/software_update_state.conf
5. Change to the folder /var/log/vmware/upgrade:
   # cd /var/log/vmware/upgrade
   
   
6. Verify that the following 3 files exist in this folder

   prechecks-succeeded
   export-succeeded
   import-succeeded

7. If these files do not exist, create them:

   # touch prechecks-succeeded
   # touch export-succeeded
   # touch import-succeeded

8. Start the appliance management service:
   $ service-control --start applmgmt
   
   
9. Login to the vCenter VAMI to validate normal operation.

Additional Information

See also the following KBs:

 

VAMI stuck at message: "Upgrade - Stage 2: Data transfer and vCenter Server setup is in progress"

 

VMware Skyline Health Diagnostics for vSphere - FAQ