Supervisor upgrade failed with errors "Component ImageRegistryUpgrade did not finish within 360 seconds timeout deadline"
search cancel

Supervisor upgrade failed with errors "Component ImageRegistryUpgrade did not finish within 360 seconds timeout deadline"

book

Article ID: 413902

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Customer firstly upgrade VC from 7.x to 8.0u3 P05 which has been successful. Post that Supervisor upgrade was attempted (from 1.27.5 -> 1.28.3), which failed with the following errors :

ImageRegistryUpgrade
error message: "Component ImageRegistryUpgrade did not finish within 360 seconds timeout deadline

EnvPropsUpgrade status

Failed to run command kubectl get vspherezones -o Jason ret=1 out = err=error: the server doesn't have a resource type "vspherezones"

Referring below KB:

https://knowledge.broadcom.com/external/article/393911/supervisor-upgrade-failed-at-envprops-co.html

After applying the KB fix, Supervisor upgrade was re-attempted, but this failed in pre-checks stage due to VKS clusters not running.

After re-triggering upgrade from dcli component ImageRegistryUpgrade completed with upgrade.

Below two components in failed status:
CsiControllerUpgrade
VMOperatorUpgrade

Error in var/log/vmware/upgrade-ctl.log

VMOperator configuration error: Component VMOperatorUpgrade failed VMOperatorUpgrade did not finish within 360 seconds timeout deadline

 

Environment

vSphere Kubernetes Service

Cause

The default timeout value for vmop upgrades needs to be increased for customer's environment due to the large number of `virtualmachineclasses` in relatively large number of `namespaces`.

Resolution

Engineering has fixed the issue by try updating the VMOP upgrade script to overwrite the default timeout seconds,this needs engineering team to confirm and execute. Please engage Broadcom support team if this kind of issue is encountered. 

This issue can't be prevented before the upgrade has taken place, but only the workaround can be applied upon the issue during the upgrade.There is no safe way to prepare the Supervisors ahead of time for this.

 

IMPORTANT NOTE: Engineering does NOT recommend doing this step before committing the upgrade. This can be done in-flight of an upgrade, or safely after upgrade has failed. There is no safe way to prepare the Supervisors ahead of time for this.

 

Additional Information

Powered by Wolken Software