"Timeout happens while waiting for task to terminate" when performing precheck for VCSA patch via VAMI UI

Products

VMware vCenter Server

Issue/Introduction

Precheck via vCenter Appliance Management Interface (VAMI) UI fails with an error:
```
Timeout happens while waiting for task to terminate
```
The same error is reported when performing the precheck with a patch ISO attached to the appliance.
vCenter Server (VCSA) is configured to use a proxy server to connect to the public internet.

In /var/log/vmware/applmgmt/Patchrunner.log reports:

<timestamp> applmgmt:CollectRequirements INFO extensions Executing patch hook '<module 'applmgmt' from '/storage/core/software-update/updates/8.0.3.00100/scripts/patches/payload/components-script/applmgmt/__init__.py'>:CollectRequirements' with context <patch_specs.PatchContext object at 0x7f65682c8340>.
<timestamp> applmgmt:CollectRequirements INFO applmgmt Executing the requirements hook for Applmgmt Service
<timestamp> applmgmt:CollectRequirements INFO applmgmt Checking if a valid file based backup was taken.
<timestamp> applmgmt:CollectRequirements INFO applmgmt.file_based_backup_recovery Get the available file based backup details from vCenter
<timestamp> applmgmt:CollectRequirements INFO vapi_utils.lookup_service Creating LookupService client with URL: https://<vcsa_fqdn>:443/lookupservice/sdk
<timestamp> applmgmt:CollectRequirements INFO vapi_utils.authentication Loading certificate and key for user 'vpxd-extension' from VECS
<timestamp> applmgmt:CollectRequirements INFO vapi_utils.authentication Key saved in file: /tmp/###
<timestamp> applmgmt:CollectRequirements INFO vapi_utils.authentication Certificate saved in file: /tmp/###
<timestamp> applmgmt:CollectRequirements INFO vapi_utils.vc_service Creating VcService client on host <vcsa_fqdn>

/var/log/vmware/applmgmt/applmgmt.log contains the following error stack:

<timestamp> [106144]DEBUG:vmware.appliance.update.update_pending:Running update prechecks for version 8.0.3.### and component None
<timestamp> [106144]INFO:vmware.appliance.update.update_functions:Timeout happens while waiting for task to terminate
<timestamp> [106144]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.update.pending in precheck - Timeout happens while waiting for task to terminate
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
    method_result = self._invoke_int(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 249, in _invoke_int
    method_result = iface.invoke(ctx, method_id, input_value)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
    meth_output = method(**meth_args)
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/update_impl.py", line 201, in precheck
    return worker.precheck(version, component)
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_pending.py", line 280, in precheck
    result = MicroserviceConnector.waitResults(request)
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/ms_connector.py", line 177, in waitResults
    result = executeFunctionUntilSuccessOrTimeout(
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 424, in executeFunctionUntilSuccessOrTimeout
    raise Exception("Timeout happens while " + ingDescription)
Exception: Timeout happens while waiting for task to terminate

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x
VMware vCenter 9.x

Cause

The issue is related to the proxy configuration on the vCenter server

Resolution

To resolve or prevent this issue, ensure that the proxy configuration file of the VCSA has the correct format and aligns with the requirements described in How to configure proxy settings for vCenter Server.

In vCenter Server 7.0.x/8.0.x

The proxy configuration is stored in /etc/sysconfig/proxy:

# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="yes"

# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY="http://<proxy_fqdn_or_ip>:<proxy_port>"

# Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY="http://<proxy_fqdn_or_ip>:<proxy_port>"

# Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY=""

# Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY=""

# Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY=""

# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""

# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1"

In vCenter 9.x

The proxy configuration is stored as JSON in /var/lib/vmware-envoy-system-proxy/config.json:

{
    "http_proxy": {
        "scheme": "<proxy_url_schema>",                                    // example: http
        "host": "<proxy_fqdn_or_ip>",                                      // example: example.com, 123.123.123.123, subdomian.example.com
        "port": <proxy_port>,
        "username": "<proxy_user_if_required>",
        "password": "<proxy_password_if_required>"
    }, // if exists or null
    "https_proxy": {
        "scheme": "<proxy_url_schema>",                                    // examples: http, https
        "host": "<proxy_fqdn_or_ip>",                                      // examples: example.com, 123.123.123.123, subdomian.example.com
        "port": <proxy_port>,
        "username": "<proxy_user_if_required>",
        "password": "<proxy_password_if_required>"
    }, // if exists or null,
    "ftp_proxy": {
        "scheme": "<proxy_url_schema>",                                    // example: ftp
        "host": "<proxy_fqdn_or_ip>",                                      // example: example.com, 123.123.123.123, subdomian.example.com
        "port": <proxy_port>,
        "username": "<proxy_user_if_required>",
        "password": "<proxy_password_if_required>"
    }, // if exists or null
    "no_proxy": ["<proxy_exclusions_if_desired>"]
}

Notes:
For vCenter Server 9.x the proxy configuration must have a valid JSON format.
CIDR proxy entries have to be manually added to the proxy configuration files, the VAMI UI does not permit CIDR additions.

Further Troubleshooting steps:

Reboot the VCSA in order to make sure any recent changes to the configuration are picked up by the operating system
Try disabling the proxy and again reboot the VCSA.
If traffic from the VCSA to dl.broadcom.com is routed through a firewall, make sure to whitelist the required IP adresses. See Public IP address list for VMware Update Manager (IP Addresses of dl.broadcom.com) for details.
Use offline methods to patch the VCSA, for example with a patch ISO or an offline depot. See Patching/Updating vCenter Server through vCenter Server Appliance Management Interface (VAMI) for more information.