Constant restart systemd-networkd on supervisor

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Seeing systemd-network service on the supervisor constantly restarting. Which can cause latency issues when you kubectl commands from supervisor. You might also see  packet loss of around 20% when pinging to supervisor control plane vm's VIP

Mar 11 14:11:53  systemd-networkd[714944]: eth0: Setting routes
Mar 11 14:11:53  systemd-timesyncd[714637]: Initial synchronization to time server <IP> (<IP>).
Mar 11 14:11:53  systemd-networkd[714944]: eth0: Routes set
Mar 11 14:11:53  systemd-networkd[714944]: eth0: link_check_ready(): dynamic addresses or routes are not configured.
Mar 11 14:11:53  systemd[1]: Starting Wait for Network to be Configured...
Mar 11 14:11:53  systemd[1]: systemd-networkd-wait-online.service: Succeeded.
Mar 11 14:11:53  systemd[1]: Stopped Wait for Network to be Configured.
Mar 11 14:11:53  systemd[1]: Starting Wait for Network to be Configured...

 

Testing kubctl command like below shows the length of time take for the command when networkd service down.

kubectl get clusters -v 9

I0401 12:00:07.396186 27549 loader.go:374] Config loaded from file: /home/<config location>/.kube/config
I0401 12:00:07.411739 27549 round_trippers.go:466] curl -v -XGET -H "User-Agent: kubectl/v1.25.6+vmware.wcp.2 (linux/amd64) kubernetes/fb5caca" -H "Authorization: Bearer <masked>" -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" 'https://<IP>/apis/cluster.x-k8s.io/v1beta1/namespaces/<namespace>/clusters?limit=500'
I0401 12:00:18.740807 27549 round_trippers.go:510] HTTP Trace: Dial to tcp:<IP>:443 succeed
I0401 12:00:18.751947 27549 round_trippers.go:553] GET https://<IP>:443/apis/cluster.x-k8s.io/v1beta1/namespaces/<namespace>/clusters?limit=500 200 OK in 11340 milliseconds
I0401 12:00:18.752089 27549 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 11328 ms TLSHandshake 5 ms ServerProcessing 5 ms Duration 11340 ms

In /var/log/update-controlller/sync.log you would see.

2025-03-13T15:44:20.234Z DEBUG network_setting: Restarted resolved and networkd.

Environment

vSphere 8.0

Cause

Domains configured are not in alphabetical order.

Can be confirmed in logs

network/10-eth0.network

[Match]
Name = eth0

[Network]
Address =<IP address>/24
Gateway = <Gateway ip>

DNS = <DNS IP> <DNS IP>
Domains =<B.Domain > < C.Domain > < A.Domain >
NTP = <NTP IP 1 > <NTP IP 2>

 

When search domains are configured with vSphere Supervisor, search domains in non alphabetical order. 

Can also be confirmed from UI Workload Management -> Network -> Management Network ->
Search Domains

 

Resolution

Search domains need to be in alphabetical order. 

Change the Management Network Settings on a Supervisor