Supervisor Cluster update fails pre-check with "Error occurred while processing HTTP response from Upgrade Compatibility Service"

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

While attempting to update the Supervisor cluster, the pre-check fails with the error:

Error occurred while processing HTTP response from Upgrade Compatibility Service: Invalid Response from UCS. StatusCode:502

The same error can be observed in the wcpsvc.log:
 

Failed to extract UCS response for cluster domain-c1006: err Invalid Response from UCS. StatusCode:502

Reviewing journalctl -xe on the supervisor node being upgraded shows the following error noting out of memory condition on UCS:

Mar 2 11:55:06 422a5ba93b8958065ffde5de57282153 kernel: upgrade-compati invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999
Mar 2 11:55:06 422a5ba93b8958065ffde5de57282153 kernel: upgrade-compati cpuset=cri-containerd-c2fa7371f2141781bc23d97b75620050347e9b6fa71dc0c7a5f7aee83848ad10.scope mems_allowed=0
Mar 2 11:55:06 422a5ba93b8958065ffde5de57282153 kernel: CPU: 11 PID: 24379 Comm: upgrade-compati Tainted: G E 4.19.164-3.ph3-esx #1-photon
Mar 2 11:55:06 422a5ba93b8958065ffde5de57282153 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020

Environment

VMware vSphere 7.0 with Tanzu

Cause

The WCP service is receiving an invalid response while trying to reach out to UCS pod to check upgrade compatibility. 

Check the status of the UCS pods on Supervisor cluster

# kubectl get pods -n vmware-system-ucs

Verify that the Pods was stable during the update phase

# kubectl describe pod <UCS-POD> -n vmware-system-ucs

If the Pod terminated because it ran out of memory, then the memory can be increased.

UCS pods caches the VirtualMachinesImages and the memory limit may not be sufficient in pre 7.0.3 releases.

In 7.0.3, the memory limit has been set to 600MB to avoid this scenario.

Resolution

Memory limit on UCS Pod is resolved in 7.0.3

Workaround:
If UCS pod has terminated due to lack of memory, increase memory limit using the following procedure:

1. SSH to one of the Supervisor Cluster VMs and edit the upgrade-compatibility-service deployment. Set the memory limit to 600 Mi

# kubectl edit deploy upgrade-compatibility-service -n vmware-system-ucs 

:
        resources:
          limits:
            cpu: "1"
            memory: 600Mi
          requests:

2. If memory is already sufficient, restart the UCS Pods:

# kubectl get pods -n vmware-system-ucs
# kubectl delete pod <UCS-POD> -n vmware-system-ucs