Troubleshooting Linux Guest OS "soft lockup" errors due to ESXi Storage VMkernel IP conflicts

Products

• VMware vSphere ESXi

Issue/Introduction

Multiple Linux Virtual Machines (VMs) running on a specific ESXi host report errors in the guest OS kernel logs:

• watchdog: BUG: soft lockup - CPU#X stuck for Xs

In the ESXi host logs, the following indicators are present:

In the var/run/log/vmkernel.log: 

• [YYYY:MM:DDTHH:MIN:SEC] In(182) vmkernel: cpu[ID]:[ID])Tcpip_Vmk: 123: arp: [##:##:##:##:##:##] is using my IP address [192.0.2.1] on vmk2!
[YYYY:MM:DDTHH:MIN:SEC] In(182) vmkernel: cpu[ID]:[ID])Tcpip_Vmk: 123: arp: [##:##:##:##:##:##] is using my IP address [192.0.2.1] on vmk2!
.....
[YYYY:MM:DDTHH:MIN:SEC] In(182) vmkernel: cpu[ID]:[ID])Tcpip_Vmk: 123: arp: [##:##:##:##:##:##] is using my IP address [192.0.2.1] on vmk2!




In the var/run/log/vobd.log: Reports duplicate IP detection events:

• [YYYY:MM:DDTHH:MIN:SEC] In(14) vobd[[ID]]: [netCorrelator] [ID]us: [vob.net.vmknic.ip.duplicate] Duplicate IP address detected for [192.0.2.1] on interface vmk#, current owner being [##:##:##:##:##:##]
[YYYY:MM:DDTHH:MIN:SEC] In(14) vobd[[ID]]: The event ([esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for [192.0.2.1] on interface vmk#, current owner being [##:##:##:##:##:##].) was sent immediately to hostd;
[YYYY:MM:DDTHH:MIN:SEC] In(14) vobd[[ID]]: [netCorrelator] [ID]us: [vob.net.vmknic.ip.duplicate] Duplicate IP address detected for [192.0.2.1] on interface vmk#, current owner being [##:##:##:##:##:##]
[YYYY:MM:DDTHH:MIN:SEC] In(14) vobd[[ID]]: [netCorrelator] [ID]us: [vob.net.vmknic.ip.duplicate] Duplicate IP address detected for [192.0.2.1] on interface vmk#, current owner being [##:##:##:##:##:##]
[YYYY:MM:DDTHH:MIN:SEC] In(14) vobd[[ID]]: The event ([esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for [192.0.2.1] on interface vmk#, current owner being [##:##:##:##:##:##].) was sent immediately to hostd;

In the var/run/log/vmkwarning.log: Reports severe performance degradation on NFS or storage volumes:

• [YYYY:MM:DDTHH:MIN:SEC] Wa(180) vmkwarning: cpu[ID]:[ID])WARNING: NFS: 5927: NFS volume [VOLUME_NAME] performance has deteriorated. I/O latency increased from average value of [VALUE](us) to [VALUE](us).
[YYYY:MM:DDTHH:MIN:SEC] Wa(180) vmkwarning: cpu[ID]:[ID])WARNING: NFS: 5965: NFS volume [VOLUME_NAME] performance has deteriorated. I/O latency increased from average value of [VALUE](us) to [VALUE](us). Exceeded threshold 10000(us)
[YYYY:MM:DDTHH:MIN:SEC] Wa(180) vmkwarning: cpu[ID]:[ID])WARNING: NFS: 5965: NFS volume [VOLUME_NAME] performance has deteriorated. I/O latency increased from average value of [VALUE](us) to [VALUE](us). Exceeded threshold 10000(us)
  
  

Cause

An IP conflict exists on the network for the VMkernel interface used for storage traffic. This duplication results in intermittent MAC/IP routing ambiguity, leading to storage timeouts that trigger the Guest OS kernel watchdog.

Resolution

To resolve this issue, the duplicate IP conflict must be cleared at the network layer.

1. Identify the Conflict:

   1. Note the MAC address reported in the vobd.log (##:##:##:##:##:##)
   2. Search the vCenter inventory for any VM or VMkernel port assigned this MAC address.
   3. Consult with the networking team to locate the physical switch port learning that MAC address.

2. Remediate the IP Assignment:

   1. Change the IP address of the conflicting device or the affected VMkernel interface to a unique value.

3. Verify Storage Stability:

   1. Once the IP conflict is resolved, monitor vmkwarning.log to ensure NFS/Storage latency returns to normal levels.
   2. Monitor the Linux guest VMs to confirm the "soft lockup" messages have ceased.

Additional Information

Duplicate IP address on ESXi vmknic interface (VMware vSphere ESXi)

Changing the ESXi server's management IP address

Soft lockup messages from Linux kernel running in an SMP-enabled virtual machine

Error: "kernel: BUG: soft lockup - CPU#Y stuck for Xs" within VM