VCSA patching to 8.0 U2 is stuck at 93% for extended time when VC has limited/no internet access

Products

• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• Patching of vCenter Server Appliance (VCSA) to 8.0 Update 2 is stuck at 93% for few hours and completes eventually
• VCSA with limited/no internet connectivity is impacted with this issue
• /var/log/vmware/applmgmt/update_microservice.log:

2023-10-18 15:14:23,120 - 34391 -        update_script::           collectRequirements: 495 -    DEBUG - Source VCSA version = 8.0.1.00200
2023-10-18 15:14:23,120 - 34391 -        update_script::           collectRequirements: 500 -     INFO - Target VCSA version = 8.0.2.00000
.
2023-10-18 15:20:37,336 - 45585 -     functions_target::              createPhotonRepo: 270 -     INFO - appliance-photon repo created successfully
2023-10-18 15:20:37,336 - 45585 -     functions_target::                    runCommand:  79 -    DEBUG - Invoking command [/bin/bash --login -c /usr/bin/photon-upgrade.sh --skip-update --assume-yes --upgrade-os --repos=appliance-photon --install-all
.
.
2023-10-18 16:51:04,634 - 54435 -        update_script::_executePatchRunnerAndSupplyUserData: 132 -    DEBUG - Patch command patch completed successfully
2023-10-18 16:51:11,472 - 45585 -    update_b2b_target::                       install:2820 -    DEBUG - Installation completed successfully!

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 8.0 U2

Cause

The upgrade to 8.0U2 involves OS upgrade from Photon3 to Photon4 and this process is attempting to connect to photon online repository for OS RPMs upgrade. In a situation VCSA is unable to access repositories, there are multiple retries which is the reason for the progress to be stuck at 93% for an extended timeline.

Resolution

This issue is fixed in vCenter Server 8.0 U2b (23319993).

Workaround:
In order to workaround the same you can try any of the below options

Option 1:

In case the patching is in progress, the recommendation is to wait if extended downtime is available as the process will complete.

Option 2:

Alternatively, execute the below steps to expedite the patching process:

• Log in to the host via Host client on which VCSA is registered
• Log in to the VCSA virtual machine via Direct Console and enable ssh. Refer to the steps under Enable Access to the Appliance Bash Shell
• Log in to VCSA via ssh
• Download the attached script and copy to the VCSA using SCP. In case of failure to access, refer to Error when uploading files to vCenter Server Appliance using WinSCP
• Navigate to the script location and and execute the below commands

chmod +x /<script location>/disable-repos.sh
/<script location>/disable-repos.sh
 

Option 3:

• Restore the VC from a valid backup or revert the snapshot of the vCenter Server before proceeding. In case of Enhanced Linked Mode (ELM), all the nodes in ELM needs to be reverted/restored at the same state to avoid any replication conflicts.
• Download the attached script and copy to the VCSA using SCP
• Log in to the VCSA via ssh
• Navigate to the script location and and execute the below command

chmod +x /<script location>/disable-repos.sh
/<script location>/disable-repos.sh

• Proceed with the upgrade via VAMI or Full Patch ISO

Note: In case if the script fails to execute, use the below sed command to convert it to linux line feed characters

sed -i -e 's/\r$//' /<script location>/disable-repos.sh