vCenter patch upgrade to 8.0.U3d upgrade fails with "Timeout happens while waiting for task to terminate" and "INTERNAL_SERVER_ERROR"

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

• During vCenter upgrade to version 8.0.U3d (Build 24322831) through SDDC Manager the task fails after some minutes with this error in 
  
  lcm-debug.log:
  
  2024-11-12T10:59:18.777+0000 DEBUG [vcf_lcm,67333506f96af74c3c725b658e05224e,6ce1] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-5] Stage details: Stage(id=174:4, name=null, description=null, type=VCENTER_UPGRADE_PRECHECK, order=4, startTime=null, endTime=null, duration=null, status=FAILED, errorType=RECOVERABLE, errorCode=null, errorDescription=Reason is: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = vapi.method.invoke.exception,
    defaultMessage = Error in method invocation Timeout happens while waiting for task to terminate,
    args = [Timeout happens while waiting for task to terminate],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = INTERNAL_SERVER_ERROR
}. , remediation=null, resources=null, errors=null, subtaskId=174, subtaskType=UPGRADE, taskId=5005fbea-7f4b-43e2-bb9e-55677fb889b4, taskType=com.vmware.vcf.lcm.upgrade)
  
  
• The following error can be seen in SDDC Manager: 

 

Environment

VMware Cloud Foundation 5.x

 

Cause

• A timeout for the vCenter update process causes the issue:

 

 /var/log/vmware/applmgmt/applmgmt.log:

2024-11-12T10:59:07 AM UTC [11163]ERROR:vmware.appliance.update.update_functions:Can't read JSON file /storage/core/software-update/updates/8.0.3.00400/precheck_results [Errno 2] No such file or directory: '/storage/core/software-update/updates/8.0.3.00400/precheck_results'
2024-11-12T10:59:07 AM UTC [11163]DEBUG:vmware.appliance.update.update_functions:Exception happens while waiting for task to terminate: FileNotFoundError(2, 'No such file or directory')
2024-11-12T10:59:07 AM UTC [11163]INFO:vmware.appliance.update.update_functions:Timeout happens while waiting for task to terminate
2024-11-12T10:59:07 AM UTC [11163]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.update.pending in precheck - Timeout happens while waiting for task to terminate
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
    method_result = self._invoke_int(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 249, in _invoke_int
    method_result = iface.invoke(ctx, method_id, input_value)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
    meth_output = method(**meth_args)
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/update_impl.py", line 201, in precheck
    return worker.precheck(version, component)
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_pending.py", line 280, in precheck
    result = MicroserviceConnector.waitResults(request)
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/ms_connector.py", line 177, in waitResults
    result = executeFunctionUntilSuccessOrTimeout(
  File "/usr/lib/applmgmt/update/py/vmware/appliance/update/update_functions.py", line 424, in executeFunctionUntilSuccessOrTimeout
    raise Exception("Timeout happens while " + ingDescription)
Exception: Timeout happens while waiting for task to terminate
2024-11-12T11:00:01 AM UTC [11163]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.system.time, operation_id: get

Resolution

Workaround:

• Please retry the upgrade from SDDC Manager until the update completes successfully. Since this is caused by a timeout issue, several attempts may be needed.