Upgrade to 8.0U3 results in Supervisor Control Plane upgrade failure due to network blocking traffic from vCenter to the Supervisor Docker image registry port

Products

• VMware vSphere with Tanzu

Issue/Introduction

After an upgrade to vCenter 8.0U3, attempts to create new service namespaces and upgrade/install tkg-service packages fail. 

From the vCenter server, we can see the following in wcpsvc.logs

2024-10-30T15:57:33.540Z debug wcp [registry/image_checker.go:26] [opID=CoreServiceController] Check image availability of 'tkg-svs/package/tkg-service:3.1.1' in registry 'http://localhost:1080/external-cert/http1/<redacted supervisor control plane node IP address>/5000' > using URL 'http://localhost:1080/external-cert/http1/<redacted supervisor control plane node IP address>/5000/v2/tkg-svs/package/tkg-service/manifests/3.1.1'
2024-10-30T15:57:33.553Z debug wcp [registry/image_checker.go:34] [opID=CoreServiceController] status 503 Service Unavailable returned from HTTP request to check image availability of 'tkg-svs/package/tkg-service:3.1.1'
2024-10-30T15:57:33.553Z error wcp [coreservice/imgpkg.go:114] [opID=CoreServiceController] imgpkg (docker-registry.kube-system.svc:5000/tkg-svs/package/tkg-service:3.1.1) is not available in registry http://localhost:1080/> external-cert/http1/<redacted supervisor control plane node IP address>/5000.
2024-10-30T15:57:33.553Z debug wcp [controller/core_service_controller.go:311] [opID=CoreServiceController] Skipping auto-installation because there are no compatible versions for service tkg.vsphere.vmware.com

In troubleshooting the issue, we can see TCP handshake completes but TLS handshake never does. 

Environment

vCenter 8.0U3
This was observed on Palo Alto firewalls, though this can affect any firewall.

Cause

An transport layer firewall rule is possibly blocking traffic between the vCenter and the Supervisor Control Plane Docker image registry. The Docker image registry should be listening on port 5000 each Supervisor control Plane node. 

 

Resolution

Allow traffic to pass unrestricted between the vCenter and Supervisor Control Plane nodes. 

Additional Information

You can find the required ports listed in the VMware by Broadcom VMware Ports and Protocols page.