Applying patch version 8.0 U2c to VMware vCenter Server fails with error: Error in upgrading all packages to latest versions (tdnf error code:245)

Products

• VMware vCenter Server 8.0

Issue/Introduction

Applying patch version 8.0 U2c to VMware vCenter Server fails with error:

Error(1525) : rpm transaction failed
Error in upgrading all packages to latest versions (tdnf error code: 245)

Cause

• The update_microservice.log shows the following entries:
  
  Error(1525) : rpm transaction failed
  Error in upgrading all packages to latest versions (tdnf error code: 245).
  Original list of RPMs and RPM DB are stored in /tmp/photon-upgrade-pp1,  please provide contents of that folder along with system journal   logs for analysis; these logs can be captured using command-
   # /usr/bin/journalctl -xa > /tmp/photon-upgrade-pp1/journal.log
   Cannot continue. Aborting.
  ]
  <yyyy-mm-dd> <hh:mm:ss>,839 - 132241 -   dbfunctions_target::                    executeDML:  56 -    DEBUG - Executing Query {UPDATE install_progress SET Subphase = ?,          end_time = DATETIME('now'), status = ?          WHERE Phase = ? } with parameters ('Install Photon RPM transaction failed.', 'failed', 'Installing RPMs')
• The update_microservice.log does not state which RPM failed, so checking the journalctl logs shows the following entries:
  

  less journalctl_-b--0.txt | grep -i rpm | egrep failure

  <date> <hh:mm:ss> [RPM][143675]: install grub2-pc-2.06-11.ph4.x86_64: failure

• We see these messages in dmesg
  
  <date> hh:mm:ss kernel: EXT4-fs error (device dm-0): ext4_lookup:1751: inode #2626896: comm tdnf: iget: checksum invalid
  <date> hh:mm:ss kernel: EXT4-fs error (device dm-0): ext4_lookup:1751: inode #2626896: comm tdnf: iget: checksum invalid

This shows filesystem issues on the root partition and can cause the rpm install failures.

Resolution

• Run a filesystem check on the root partition.
• Take a snapshot or backup of the VCSA before proceeding.
• Load the grub menu as described in the KB https://knowledge.broadcom.com/external/article?legacyId=2147144 until step 4 and then add "fsck.mode=force fsck.repair=yes" as shown in the screen shot below and hit F10.
  
  
  
  
• After the VMware vCenter Server boots, you can attempt the patch again and the patch should go through.
• If the patch fails with the same error, collect the following to investigate further:
  • Logs for the ESXi host that the VCSA VM is running on.
  • vCenter logs.
  • Run this command on VCSA and save output #  lsblk -OJ