vCenter Server upgrade failure from corrupt ISO

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server upgrades may fail due to corrupted or incomplete ISO files. Checksum verification ensures the integrity of the upgrade ISO file before attempting the upgrade process. This procedure is essential when upgrade failures occur with errors indicating file corruption or checksum mismatches.

Common symptoms that may indicate checksum issues include:

• Upgrade failures during the staging phase
• Error messages mentioning checksum mismatches
• Unexpected upgrade termination
• Package extraction failures

Log Snippets:

• /var/log/vmware/applmgmt/update_microservice.log

  • YYYY-MM-MM HH:MM:SS, - 36754 - rpmfunctions:: isChecksumValid: 356 -    ERROR - Checksum does not match, expected [a59bc23205b27e18f8ed1a1e9dde2f7553c9bdc214cbca15b3df2070bf1a21d6], found [c6455fa92df1e39f27045547f2ee4878ce50c09fd868eb508a3f1597598fed3e] 

Environment

• vCenter Server 7.x
• vCenter Server 8.x
• Linux-based systems for ISO verification

Cause

ISO checksum mismatches can occur due to:

• Incomplete download of the upgrade ISO
• Network corruption during download
• Storage corruption on the local system
• Using an incorrect or modified ISO file

Resolution

• 1. Check for Checksum Errors in Logs
     1. Access the vCenter Server Appliance via SSH as the root user.
     2. Review the update microservice log for checksum-related errors:
        

        grep -i checksum /var/log/vmware/applmgmt/update_microservice.log

     3. Look for error messages similar to:
        • "Checksum mismatch detected"
        • "SHA256 verification failed"
        • "File integrity check failed"
          
          
  2. Obtain Correct Checksum Value
     1. Visit the official VMware/Broadcom download page for your vCenter version.
     2. Locate the checksum information (usually SHA256) provided alongside the ISO download.
     3. Record the official checksum value for comparison.
        
        
  3. Verify ISO Checksum
     1. Navigate to the directory containing the upgrade ISO file.
     2. Calculate the SHA256 checksum of your ISO file:

        sha256sum /path/to/vcsa-upgrade.iso

         Example:
        

        sha256sum /tmp/VMware-VCSA-all-8.0.2-22617221.iso

     3. Compare the calculated checksum with the official checksum from the VMware/Broadcom download page.
        
        
  4. Resolve Checksum Mismatches
         
              If checksums do not match:
        
     1. Delete the corrupted ISO file:
        

        rm /path/to/vcsa-upgrade.iso

     2. Re-download the ISO file from the official Broadcom download portal.
     3. Verify the new download using the checksum verification process above.
     4. Ensure the checksums match before proceeding with the upgrade.
            
        If checksums match:
           
        The ISO file is valid and the issue may be related to other factors
        Proceed with other troubleshooting steps for the upgrade failure
        
        
  5. Additional Verification Steps
     1. Check available disk space to ensure sufficient room for the upgrade:
        

        df -h

     2. Verify file permissions on the ISO:

        ls -la /path/to/vcsa-upgrade.iso

     3. Test ISO integrity by attempting to mount it:

        mkdir /tmp/iso_test
        mount -o loop /path/to/vcsa-upgrade.iso /tmp/iso_test
        ls /tmp/iso_test
        umount /tmp/iso_test

  Verification

  After completing checksum verification:

  1. Confirm the calculated checksum matches the official VMware/Broadcom checksum
  2. Verify the ISO file can be mounted successfully
  3. Ensure sufficient disk space is available for the upgrade process
  4. Attempt the vCenter upgrade process with the verified ISO
  5. Make sure that the previously staged ISO is unstaged.
     1. Take an SSH to vCenter
     2. Access the Appliance shell (appliancesh) and not Bash Shell to perform the patch
        1. When in the appliance shell, the prompt will begin with: command>
        2. Check if there are any staged package already:
           • software-packages list --staged
        3. If the above output returns the previous ISO staged, unstage it with the below command:
           • software-packages unstage
  6. Proceed ahead with CLI or GUI patching of vCenter

Additional Information

• Always use the official Broadcom download portal for ISO files
• SHA256 is the standard checksum algorithm used for VMware ISO verification
• Large ISO files may take several minutes to calculate checksums
• Consider using download managers or resume-capable tools for large ISO downloads to prevent corruption