VCSA patching to 8.0U2 or Higher version fails with "Internal error occurs during execution of update process. Pre-Install failed for vpxd:Expand"

Products

• VMware vCenter Server 8.0
• VMware Cloud Foundation 5.x

Issue/Introduction

Symptoms:

• vCenter Server Appliance (VCSA) patching to 8.0 U2 or Higher version fails with "Internal error occurs during execution of update process. Pre-Install failed for vpxd:Expand"
• In /var/log/vmware/applmgmt/PatchRunner.log we see below entries.

vpxd:Expand INFO vmware_b2b.patching.utils.reporting_utils Setting global ReportingFactory with identifier - vpxd:Expand
vpxd:Expand INFO extensions Found patch hook <module 'vpxd' from '/storage/seat/software-update39az1wwa/stage/scripts/patches/payload/components-script/vpxd/__init__.py'>:Expand'
vpxd:Expand INFO extensions Executing patch hook '<module 'vpxd' from '/storage/seat/software-update39az1wwa/stage/scripts/patches/payload/components-script/vpxd/__init__.py'>:Expand'
with context <patch_specs.PatchContext object at 0x7fb9a9382710>.
vpxd:Expand INFO vpxd Vpxd expand phase started
vpxd:Expand INFO vpxd Placed B2B marker
vpxd:Expand INFO vpxd Placed temp NDU marker
vpxd:Expand INFO vpxd Copied vpxd cfg files to tmp
vpxd:Expand INFO vpxd Completed vpxd.cfg.tmp modification
vpxd:Expand ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpxd:Expand' failed.
Traceback (most recent call last):
  File "/storage/seat/software-update39az1wwa/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
   File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 301, in perform_request
    raise Exception("Got response %s: %s\n%s" %
Exception: Got response 503: content-length: 1
 

• Reference Screenshot

 

Environment

VMware vCenter Server 8.0.2

VMware Cloud Foundation 5.1.0
VMware Cloud Foundation 5.1.1

Cause

This error is seen if the start operation using service-control enters a rare race condition during the patch workflow

Resolution

This issue is resolved in VMware vCenter Server 8.0 Update 3 release and VMware Cloud Foundation 5.2.0

Workaround:

In order to resolve the issue, perform one of the below

1. Log in to the VCSA using ssh

2. Start the services manually using the below commands

service-control --start lookupsvc sts

3. Access the VAMI (vCenter Appliance Management Interface) and click Resume vCenter update

If Resume vCenter update is not available, select Rollback to the previous version and then restart the Patching via VAMI.

Note: In case if the above steps is not recovering from the failed state, proceed with restore of the vCenter server. In case if vCenter is in ELM (enhanced linked mode), then all the nodes must be restored from backup. If the failed vCenter server is part of VCF instance, engage technical support for assistance with recovery.

In case that the vCenter Server update was initiated by SDDC Manager, the following additional steps are required.

4. Wait the update vCenter Server by SDDC Manager is failed with 2 hour timeout.

5. Retry update vCenter Server by SDDC Manager.