vCenter Upgrade Fails with "No such file or directory: patching_backup

Products

VMware vCenter Server

Issue/Introduction

When attempting to upgrade vCenter Server, the process fails with an error message similar to:

[Errno 2] No such file or directory: '/storage/seat/software-update*/stage/update/patching_backup_config.json'

This error typically occurs during the staging phase of the upgrade process and prevents the upgrade from completing successfully.

Environment

- VMware vCenter Server 7.0 or newer
- Upgrading to a newer version

Cause

This error can occur due to several reasons:

1. A previous failed upgrade attempt left the system in an inconsistent state.
2. Corruption in the RPM database.
3. Issues with SSL certificates or trust.
4. Incorrect VMware Directory (vmdir) Domain Functional Level.
5. Insufficient storage space, particularly in the log partition.
6. Checksum mismatch in downloaded packages or ISO file.

Resolution

Follow these steps to resolve the issue and successfully complete the vCenter upgrade:

Option #1

1. Unstage the update from vCenter server (refer to document below bullet point #3)..

2. Stage the update again.

3. Run the update.

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-FF533442-66F0-4797-976D-1DA99102DD0A.html

Option #2 - if Option #1 does not work:

Take a powered-off snapshot of the vCenter Server virtual machine before proceeding.
Access the vCenter Server Appliance (VCSA) via SSH as the root user.
Rebuild the RPM database:
1. Create a backup directory for the RPM database:
  - ```
  mkdir /storage/core/rpmdbBackup/
```
2. Move the current RPM database files:
  - ```
  cd /var/lib/rpm
  mv __db* /storage/core/rpmdbBackup/
```
3. Rebuild the RPM database:
  - ```
  rpm --rebuilddb
```
Restart the applmgmt service:
- - ```
  service-control --restart applmgmt
```

Run the LSDoctor tool to address potential SSL and trust issues:

Fix stale services:

python /usr/lib/vmware-lookupsvc/tools/lsdoctor.py --stalefix

Fix SSL trust mismatches:

python /usr/lib/vmware-lookupsvc/tools/lsdoctor.py --trustfix

Verify and set the correct VMware Directory (vmdir) Domain Functional Level:
1. Follow the steps in VMdir enters failure state after upgrading vCenter Server to 8.0 U1.
Check available storage space, especially in the log partition:
1. Follow the vCenter Server Appliance disk space is full to resolve any full storage space issue on the vCenter VM.
Check for any certificate issues:
1. Follow Using the VCF Diagnostic Tool for vSphere (VDT)
2. Replace certificates indicated by the tool.

Remove the failed update state file:

rm /etc/applmgmt/appliance/software_update_state.conf

Restart the applmgmt service again:
- - ```
  service-control --restart applmgmt
```
Check for checksum issues:
1. Review the `/var/log/vmware/applmgmt/update_microservice.log` file for checksum mismatch errors.
2. If checksum errors are found, re-download the upgrade ISO from the VMware website.
  1. Verify the ISO checksum using the command:
    - ```
    sha256sum /path/to/vcsa-upgrade.iso
```
2. Compare the output with the checksum provided on the VMware download page.
Access the vCenter Server Appliance Management Interface (VAMI) and attempt the upgrade process again:
1. Stage the update.
2. Install the update.

Additional Information

- Always take a snapshot or backup of the affected vCenter Server before attempting an upgrade or applying these fixes.
- For persistent issues, consult the following log files for more detailed error information:
- /var/log/vmware/applmgmt/update_microservice.log
- /var/log/vmware/applmgmt/software-packages.log
- For linked-mode vCenter Server environments, perform these steps on each vCenter Server in the topology, starting with the one that experienced the upgrade failure.