Embedded Harbor reports healthcheck failure after host enters MaintenanceMode in VKS
search cancel

Embedded Harbor reports healthcheck failure after host enters MaintenanceMode in VKS

book

Article ID: 305332

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere Kubernetes Service

Issue/Introduction

  • vCenter GUI shows errors similar to the following on the Embedded Harbor Registry after the host on which the PODVM's were initially deployed is placed into Maintenance Mode:

    • "Harbor registry harbor-#### on cluster domain-c# is unhealthy. Reason: Could not determine the reason causing Harbor status to be unhealthy"  OR 
    • "Harbor registry harbor-#### on cluster domain-c# is unhealthy. Reason: failed to get harbor health: Get https://<IP>:443: connect: network is unreachable"

      Where <IP> is the external LoadBalancer Service IP address of Harbor, used to connect to Harbor UI

  • The VMware System Registry Controller Manager logs report failure to check health on jobservice, similar to:

    # kubectl logs -n vmware-system-registry vmware-registry-controller-manager-######-##### -c admin-agent

    level=info msg="harbor registry status is updated, status: {v1.10.8 <IP> {registry_ready Harbor registry harbor-#### is deployed in namespace vmware-system-registry-####} {unhealthy Harbor component jobservice is unhealthy: failed to check health: Get http://harbor-1597465241-harbor-jobservice/api/v1/stats: dial tcp <IP>:80: i/o timeout;}

  • Testing 10250 port connectivity to ESXi hosts in the workload cluster over Supervisor NICs shows a failure:

    • curl -v telnet://<ESXi_Management_IP>:10250 --interface eth0
    • curl -v telnet://<ESXi_Management_IP>:10250 --interface eth1

  • SSH on the ESXi host to confirm that Spherelet is operational and listening on the expected port 10250:

    • Check Spherelet status:
      • /etc/init.d/spherelet status

    • Check Firewall on ESXi host to ensure ports are opened:
      • esxcli network firewall ruleset list | egrep "Name|true"
      • esxcli network firewall ruleset rule list |egrep "Ruleset|Inbound"

Environment

vSphere Kubernetes Service

Cause

The Supervisor Cluster Control Plane VM's cannot connect to ESXi over port 10250.

Resolution

Ensure port 10250 is open from Supervisor Control Plane VM's to ESXi Management Kernel IP address.

Additional Information

Embedded Harbor registry will become unuseable if port 10250 is blocked from Supervisor Cluster VM's to the ESXi management IP address. 10250 is the port on which Spherelet runs, so this can impact other PODVM operations as well.