Supervisor cluster upgrade failed at Component ImageRegistry.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Supervisor upgrades failed at component ImageRegistry and gets stuck (eg, 50%).
• We can check with the below command:

  /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py get-status | jq '.progress | to_entries | .[] | "\(.value.status) - \(.key)"' | sort
  

• From DCLI, we see :

  message: Namespaces cluster upgrade is in the "upgrade components" step.

•  In the compupgrade logs of the Supervisor we see below entries:

2024-08-30T08:10:37.990Z INFO compupgrade: Current status: {'controller_id': '422f0322b476dee571f7fd95293376b1', 'state': 'error', 'messages': [{'level': 'error', 'message': "Component ImageRegistryUpgrade failed: argument of type 'NoneType' is not iterable"}, {'level': 'error', 'message': 'Component upgrade failed.'}], 'started_at': '2024-08-30T08:01:42.946712Z', 'last_modified': '2024-08-30T08:09:10.980427Z', 'progress': {'WCPClusterCapabilities': {'status': 'upgraded', 'messages': []}, 

2024-08-30T08:45:07.298Z ERROR compupgrade: {"error": "TypeError", "message": "argument of type 'NoneType' is not iterable", "backtrace": ["  File \"/usr/lib/vmware-wcp/upgrade/compupgrade.py\", line 362, in do_upgrade_with_out_resume_failed_support\n    comp.doUpgrade(upCtx)\n", "  File \"/usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py\", line 323, in doUpgrade\n    self.updateV1alpha1ImageRegistryResources()\n", "  File \"/usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py\", line 171, in updateV1alpha1ImageRegistryResources\n    self.updateV1alpha1Resource('contentlibraries', True)\n", "  File \"/usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py\", line 193, in updateV1alpha1Resource\n    patch_list = self.getResourcePatchBody(status, resource_kind)\n", "  File\"/usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py\", line 218, in getResourcePatchBody\n    if creation_time and 'UTC' in creation_time:\n"]}
2024-08-30T08:45:07.298Z INFO comphelper: Running ['/usr/local/bin/etcdctl', 'get', '/vmware/wcp/upgrade/components/status', '--print-value-only', '--command-timeout=30s', '--dial-timeout=5s']

Environment

• vSphere with Tanzu 8.0

Cause

Upgrade of the Image-registry fails and fails to re-trigger the upgrade of component.

Resolution

To resolve the issue you can manually update the imageregistry_upgrade.py script to re-trigger the imageregistry upgrade.
This should be done on the CPVM that has the errors in /var/log/vmware/upgrade-ctl-compupgrade.log.

To update the script, follow the steps below:

1. Create a backup copy of the script:

cp /usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py /usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.old

2. Edit the script /usr/lib/vmware-wcp/objects/image-registry-operator/imageregistry_upgrade.py and go to line 221 which contains:

if 'UTC' in creation_time:

3. Replace this line with:

if creation_time and 'UTC' in creation_time:

4. Now move on to line 225 containing:

if 'UTC' in last_modified_time:

5. Replace this line with:

if last_modified_time and 'UTC' in last_modified_time:

6. Save the file and it will trigger an upgrade