New k8s node creation failed with "There is no network assigned to this virtual machine"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When scaling out the number of worker nodes, the new created VM is

• Not powered-on and said "There is no network assigned to this virtual machine."
• Associated Port Group is not attached on the VM

# Supervisor context
kubectl -n vmware-system-netop logs deployment/vmware-system-netop-controller-manager

YYYY-MM-DDThh:mm:ss.###Z stderr F E0530  1 networkinterface_controller.go:250] controllers/NetworkInterface/<VSPHERE_NAMESPACE>/<NEW_CREATED_VM_NAME> "msg"="Error reconciling NetworkInterface" "error"="exhausted all ip addresses in requested ip pools"
YYYY-MM-DDThh:mm:ss.###Z stderr F E0530  1 controller.go:317] controller/networkinterface "msg"="Reconciler error" "error"="exhausted all ip addresses in requested ip pools" "name"="<NEW_CREATED_VM_NAME>" "namespace"="<VSPHERE_NAMESPACE>" "reconciler group"="netoperator.vmware.com" "reconciler kind"="NetworkInterface"

Environment

VMware vSphere Kubernetes Service using VDS + AVI Loadbalancer network

Cause

Many stale kind:NetworkInterface entries remain in the Supervisor, resulting in exhaustion of available IP addresses.  This issue indicates that object updates within the Supervisor are not functioning correctly.

kubectl -n <VSPHERE_NAMESPACE> get networkinterfaces

NAME                                      AGE
<CLUSTER_NAME>-worker-node-###-7qpd4      93d
<CLUSTER_NAME>-worker-node-###-8w5lf      55m
<CLUSTER_NAME>-worker-node-###-cx8wz      228d
<CLUSTER_NAME>-worker-node-###-djpfj      166d
<CLUSTER_NAME>-worker-node-###-jbqpl      55m
<CLUSTER_NAME>-worker-node-###-k6p5x      166d
...
...
(You can observe that the number of registered entries is significantly higher than the actual number of nodes)

Resolution

Expiration of internal Supervisor certificates may prevent object cleanup. Refer to the KB article to verify their validity and rotate them if necessary.

KB - Replace vSphere with Tanzu Supervisor Certificates

After the Supervisor issue is resolved, the kind:NetworkInterface object will be reconciled, and the released IP address will be assigned to the newly created node.

Additional Information

• KB - TKC cluster stuck in creating state with error "exhausted all ip addresses in requested ip pools"
• kind: NetworkInterface is used when enabling VDS + AVI Loadbalancer setup, not used under NSX setup