vCenter patching fails with error "All the packages are already installed"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After a failed update/patch of vCenter attempting to re-stage the patch fails with the following error:

• Attempting to stage the patch via CLI also fails with the following error:

# software-packages stage --iso
    [YYYY-MM-DDTHH:MM:SS] : Discovering updates...
    [[YYYY-MM-DDTHH:MM:SS] : UpdateInfo: Using product version <Version Number> and build <Build Number>
    [[YYYY-MM-DDTHH:MM:SS] : Target VCSA version = <Version Number>
    [[YYYY-MM-DDTHH:MM:SS] : Stage path file doesn't exist
    [[YYYY-MM-DDTHH:MM:SS] : Staging failed. Retry to resume from the current state. Or please collect the VC support bundle.

Cause

This issue occurs due to a failed vCenter patching attempt leaving the vCenter in a state where newer RPMs for the target version have already been installed. vCenter doesn't know how to proceed for patching as during the stage operation, it downloads the newer version RPMs, but since the new RPMs are already installed, it will not complete the stage operation.

From /var/log/vmware/applmgmt/update_microservice.log below logs are available:

 

    [YYYY-MM-DDTHH:MM:SS DEBUG:vmware.appliance.update.functions:Resume needed in Stage phase

    [YYYY-MM-DDTHH:MM:SS ERROR:vmware.appliance.update.update_b2b:Database not initialised before its use.

    [YYYY-MM-DDTHH:MM:SS INFO:vmware.vherd.base.software_update:Staging failed. Retry to resume from the current state. Or please collect the VC support bundle.

 

    [YYYY-MM-DDTHH:MM:SS - task_manager:: update: 80 - DEBUG - UpdateTask: status=FAILED, progress=0, message={'id': 'com.vmware.appliance.nothing to stage', 'default_message': 'All the packages are already installed.', 'args': []}

    [YYYY-MM-DDTHH:MM:SS - update_b2b:: resumeStage:3197 - DEBUG - Returning from stage(), all packages are already installed

 

 

From /var/log/vmware/applmgmt/software-packages.log below logs are available:

 

    YYYY-MM-DDTHH:MM:SS INFO:vmware.appliance.update.update_pending:No updates found

    YYYY-MM-DDTHH:MM:SS INFO:vmware.vherd.base.software_update:Latest updates already installed on VCSA, Nothing to stage

 

    YYYY-MM-DDTHH:MM DEBUG:vmware.vherd.base.software_update:Initialized software update state

 

    Staged status = False

    Staged packages:

    {}

Installed packages:

{}

Payload packages:

{}

MetaData:

{}

Script results:

 

    YYYY-MM-DDTHH:MM:SS DEBUG:vmware.vherd.base.software_update:Stage configuration /storage/core/software-packages/staged-configuration.json not present

    YYYY-MM-DDTHH:MM:SS DEBUG:vmware.vherd.base.software_update:Destroyed software update state

Resolution

To resolve this issue please perform one of the options below.

Option1: Revert the vCenter to a snapshot when the vCenter was healthy.

Option2: Restore the vCenter from a last good backup (This option is most recommended).

Option3: If there is no vCenter snapshot / backup to roll back, follow the below steps:

NOTE: Please take a valid vCenter snapshot / backup prior to performing these steps.

• Copy the file "/var/lib/vmware/vpxd-svcs/config/dataservice.properties" to "/storage/core/software-update/updates/<Target vCenter Version>/patch_runner/vpxd-svcs/" using the below command:
  cp /var/lib/vmware/vpxd-svcs/config/dataservice.properties /storage/core/software-update/updates/<Target vCenter Version>/patch_runner/vpxd-svcs/
• Stop all the vCenter services using below command:
  service-control --stop --all
• Go to "/storage/core/software-update/updates/<vCenter version>/scripts/patches" and run the below command:
      python3 PatchRunner.py patch -d /storage/core/software-update/updates/<vCenter version>/patch_runner
• After above step completes successfully, change the "build", "version", and "name" fields manually in "/etc/applmgmt/appliance/update.conf" file.
• Then reboot the vCenter and confirm login and build number has updated successfully.