Deployment of WCP and installation of spherelet on hosts fails with "nodes \"<ESXi_HOST>.\" is forbidden: node \"<ESXi_HOST>\" is not allowed to modify node \"<ESXi_HOST>.\""

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Deployment of WCP has been initiated and is progressing

• Selecting View for the Host Config Status shows a warning stating: Configured ESXi Host as a Kubernetes Worker Node. A general system error occurred. Error message: context deadline exceeded

• The ESXi host's /var/log/spherelet.log file contains:  

2024-11-01T22:22:40Z No(5) spherelet[2106069]: time="2024-11-01T22:22:40.179001Z" level=fatal msg="nodes \"<ESXi_HOST>.\" is forbidden: node \"<ESXi_HOST>\" is not allowed to modify node \"<ESXi_HOST>.\""  (Notice the trailing dot after <ESXi_HOST>)

• When trying to take one specific host out of maintenance mode, it never finishes. 

Cause

This issue occurs when the ESXi host running the supervisor control plane VM with the VIP isn't able to determine it's fully qualified domain name (FQDN) from its hostname and domain. This can happen when the ESXi host:

• Doesn't have a domain name configured 
• Has an period at the end of its name in its /etc/hosts file

Resolution

• Set the ESXi host's domain name by selecting it in vCenter and going to Configure > TCP/IP configuration > Default > Edit > Populate the domain name field
• Verify that the ESXi host's /etc/hosts file file contains valid entries for the host, for example: <HOST_IP> host.example.com host