Updating or Staging of patch on vCenter server returns error: Error in Method invocation Timeout happens while waiting for task to terminate

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• Staging or Updating of VCSA (vCenter Server Appliance) via VAMI returns error:

Error in Method invocation Timeout happens while waiting for task to terminate

• In this vCenter log file /var/log/vmware/applmgmt/applmgmt.log - the following is observed:

YYYY-MM-DDTHH:MM:SS AM UTC [2509283]DEBUG:root:Validated user privileges in localstore or SSO
YYYY-MM-DDTHH:MM:SS AM UTC [2509283]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
YYYY-MM-DDTHH:MM:SS AM UTC [2509283]ERROR:vmware.appliance.update.update_functions:Can't read JSON file /storage/core/software-update/updates/8.0.2.00300/precheck_results [Errno 2] No such file or directory: '/storage/core/software-update/updates/8.0.2.00300/precheck_results'
YYYY-MM-DDTHH:MM:SS AM UTC [2509283]DEBUG:vmware.appliance.update.update_functions:Exception happens while waiting for task to terminate: FileNotFoundError(2, 'No such file or directory')
YYYY-MM-DDTHH:MM:SS AM UTC [2509283]ERROR:vmware.appliance.update.update_functions:Can't read JSON file /storage/core/software-update/updates/8.0.2.00300/precheck_results [Errno 2] No such file or directory: '/storage/core/software-update/updates/8.0.2.00300/precheck_results'

• The vCenter PatchRunner.log will be stuck at the below line

/var/log/vmware/applmgmt/PatchRunner.log

YYYY-MM-DDTHH:MM:SS vmidentity:CollectRequirements INFO extensions Executing patch hook '<module 'vmidentity' from '/storage/core/software-update/updates/8.0.2.00400/scripts/patches/payload/components-script/vmidentity/__init__.py'>:CollectRequirements' with context <patch_specs.PatchContext object at 0x7f5c0b767160>.
YYYY-MM-DDTHH:MM:SS vmidentity:CollectRequirements INFO vmidentity Executing Requirements Hook for VMware Identity STS Service
YYYY-MM-DDTHH:MM:SS vmidentity:CollectRequirements INFO vmidentity.utils Found configured HTTP port 80
YYYY-MM-DDTHH:MM:SS vmidentity:CollectRequirements INFO vmidentity.utils providers command output is - {"value":[]}

• sso-config command to fetch the authentication policy on vCenter is hung:

root@vcenter [ / ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
vsphere.local

root@vcenter [ / ]# sso-config.sh -get_authn_policy -t vsphere.local

Authentication policy on tenant : 'vsphere.local' :
IsPasswordAuthEnabled:  true
IsWindowsAuthEnabled:  true
IsTLSClientCertAuthnEnabled:  false
IsSecurIDAuthnEnabled:  false
revocationCheckEnabled:  true
useOCSP:  false
useCRLAsFailOver:  false
Alternative OCSP responders:
useCertCRL:  true
CRLUrl:   UndefinedConfig
Exception in thread "main" java.lang.NoSuchMethodError: com.vmware.identity.idm.ClientCertPolicy.getEnableHint()Z
        at com.vmware.identity.ssoconfig.SsoConfig.displayAuthenticationPolicy(SsoConfig.java:1064)
        at com.vmware.identity.ssoconfig.SsoConfig.get_authn_policy(SsoConfig.java:1000)
        at com.vmware.identity.ssoconfig.SsoConfig.main(SsoConfig.java:542)

Environment

VCF 5.x

VCF 9.x

vCenter server 8.x

Cause

Intermittent connectivity issues (potentially caused by a proxy) with reaching the vvs.broadcom.com endpoint

Resolution

To workaround the issue, retry the workflow and confirm forward and reverse lookup return successfully for the vCenter to the DNS server. 

Alternative method 1:  perform the below steps

1. Log in to Broadcom Support portal web page
2. Download the Full-Patch ISO for the current VCSA build from the download page. Refer to VMware vSphere downloads, VMware Converter, OEM custom images, patches and addons in the Broadcom Support Portal
3. Extract or Mount the file on a client machine
4. Copy the rpm "vmware-identity-sts-<version>-<build>.noarch.rpm" file to the affected vCenter server /tmp
5. Take a proper backup of the VCSA before proceeding
6. Manually install the rpm on the VCSA via the below sample command:
   • cd /tmp
   • rpm -Uvh --force vmware-identity-sts-8.0.2.00100-11979815.noarch.rpm
7. Restart all the services on the vCenter
   • service-control --stop --all && service-control --start --all
8. Proceed with the patching of vCenter server

Method 2: Increase the pre-check timeout value on the vCenter Server Appliance:

1. Log in to the vCenter Server via SSH using the root user.
2. Stop the Management Service: service-control --stop applmgmt
3. Modify the Pre-check Timeout Script using the vi editor:  vi /usr/lib/applmgmt/update/py/vmware/appliance/update/ms_connector.py
4. Locate the following function definition: def waitResults(cls, request, timeout=300):
5. Modify the default timeout parameter to double the wait time. Change timeout=300 to timeout=600.
6 .Save and exit the editor (:wq).
7. Restart the Management Service: service-control --start applmgmt
8. Retry the upgrade process from the VAMI.