Supervisor Cluster Upgrade Pre-Check Failure: Error occurred while processing HTTP response from Upgrade Compatibility Service: Invalid Response from UCS. StatusCode: 500

Products

• VMware vSphere Kubernetes Service
• Tanzu Kubernetes Runtime

Issue/Introduction

^{When performing a supervisor cluster upgrade, the pre-check fails with the error:
"Error occurred while processing HTTP response from Upgrade Compatibility Service: Invalid Response from UCS. StatusCode: 500."}

^{This issue typically occurs when stranded clusters are present, preventing the upgrade process from completing successfully.}

Environment

^{VMware vSphere with Tanzu}

vCenter 8.0u2 and earlier*

*This issue only occurs in Supervisor clusters which do not have TKG service installed.

Cause

The error is caused by the presence of stranded clusters in the environment, which can prevent proper communication between the Upgrade Compatibility Service and UCS.

This may result in the service returning an invalid response (StatusCode: 500).

This issue only occurs in Supervisor clusters which do not have TKG service installed.

Resolution

^{Follow the steps below to resolve the issue:}

1. ^{Check the Cluster Status:}

^{On the vCenter Server, open a PuTTY session and run the following commands:}

• ^{Log in to the system:}

dcli +i

• ^{List the available clusters:}

com vmware vcenter cluster list

• ^{Get the cluster information for the affected supervisor domain (replace domain-cxxx with the actual supervisor domain name):}

com vmware vcenter namespacemanagement software clusters get --cluster domain-cxxx

2. ^{Verify the Paused State:}

^{While connected to the Supervisor cluster context, ensure that the cluster is in a paused state. Run the following command:}

kubectl get cluster clustername -n namespace -o json | jq '.metadata.annotations | has("tkg.tanzu.vmware.com/paused")'

3. ^{Check Annotations for KCP and MDs:}

^{Verify that both the KCP (Kubernetes Control Plane) and MDs (Machine Deployments) have the correct annotation. Use the following command to check:} 

kubectl get kcp -n namespace clustername-control-plane -o json | jq '.metadata.annotations | has("cluster.x-k8s.io/paused")'

4. ^{Verify Cluster Pause Status:}

^{Check if the cluster status is marked as "Paused":}

 kubectl get cluster clustername -n namespace -o json | grep -i pause

5. ^{Follow the KB Article for resolution:}

^{Please follow the detailed resolution steps outlined in the following KB article:
TKCS Unable to be Deleted After an Auto-Update}

Additional Information

This issue only occurs in environments which do not have TKG service installed.

Supervisor Upgrade Precheck Failure Guide