Embedded Harbor reports healthcheck failure after host enters MaintenanceMode

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:
vCenter GUI shows errors similar to the following on the Embedded Harbor Registry after the host on which the PODVM's were initially deployed is placed into Maintenance Mode:

"Harbor registry harbor-1597465241 on cluster domain-c8 is unhealthy. Reason: Could not determine the reason causing Harbor status to be unhealthy"

Or:

"Harbor registry harbor-1597465241 on cluster domain-c8 is unhealthy. Reason: failed to get harbor health: Get https://10.245.0.10:443: connect: network is unreachable"

Where 10.245.0.10 is the external LoadBalancer Service IP address of Harbor, used to connect to Harbor UI

The VMware System Registry Controller Manager logs report failure to check health on jobservice, similar to:

# kubectl logs -n vmware-system-registry vmware-registry-controller-manager-######-##### -c admin-agent

 

time="2022-05-02T17:16:03Z" level=info msg="harbor registry status is updated, status: {v1.10.8 10.245.0.10 {registry_ready Harbor registry harbor-1597465241 is deployed in namespace vmware-system-registry-1597465241} {unhealthy Harbor component jobservice is unhealthy: failed to check health: Get http://harbor-1597465241-harbor-jobservice/api/v1/stats: dial tcp 10.244.0.10:80: i/o timeout;}

Where 10.244.0.10 is the ClusterIP for the jobservice service.

Environment

VMware vSphere 7.0 with Tanzu

Cause

The vSphere with Tanzu Supervisor Cluster Control Plane VM's cannot connect to ESXi over port 10250:

Connect to Supervisor SSH:

Test 10250 port connectivity to ESXi hosts in the workload cluster over both Supervisor NICs using:

# curl -v telnet://<ESXi_Management_IP>:10250 --interface eth0
# curl -v telnet://<ESXi_Management_IP>:10250 --interface eth1

If you see this fail on either interface, the physical network needs to be checked to ensure port 10250 is allowed.

Resolution

Check the firewall to ensure port 10250 is allowed from Supervisor Control Plane VM's to ESXi Management Kernel IP address

Workaround:
None found. This requires opening ports if they are blocked

Additional Information

Impact/Risks:
Embedded Harbor registry will become unuseable if port 10250 is blocked from Supervisor Cluster VM's to the ESXi management IP address. 10250 is the port on which Spherelet runs, so this can impact other PODVM operations as well.