Unix servers experience severe slowness during patching and unexpectedly enter emergency mode.

Products

• VMware vSAN

Issue/Introduction

Your Unix servers experience severe slowness during patching and unexpectedly enter emergency mode.

You verify the issue by checking the physical uplink status on the host; the vSAN dedicated NIC is linking at 100 Mbps, which is below the supported minimum.

If you SSH with root to the effected host, and search the /var/log/vmkernel.log file, you will see log entries similar to: CMMDS: LeaderUpdateMeanRTLatency... High RT latency... 549(ms)

Environment

vSAN 8.x

Cause

A physical NIC hardware failure caused the interface to auto-negotiate down to 100 Mbps.

vSAN requires a minimum of 1 Gbps (10 Gbps+ recommended). The restricted bandwidth causes severe storage latency, preventing timely I/O acknowledgment. This forces the Guest OS into emergency mode due to storage timeouts.

 

Resolution

You must resolve the hardware fault to restore stability.

1. Replace the failed Network Interface Card (NIC) via your hardware vendor.

2. Once replaced, verify the link speed restores to 10 Gbps (or the rated speed).

3. Reboot the affected VMs to exit emergency mode.

Additional Information

Also, review network latency kb on vSAN for further details.