vSANFS NFS Server stopped responding until NFS Client re-create TCP session

Products

• VMware vSAN

Issue/Introduction

According to the packet capture, you might see below flows.

// This is the last reply from NFS Server.
####80 284.054595 ServerIP → ClientIP NFS 310 V4 Reply (Call In 156879) GETATTR
// Client got any response from here.
####85 284.055923 ClientIP → ServerIP NFS 262 V4 Call GETATTR FH: 0x6d6856ea
// Someone claimed ClientIP from different MAC Addres, which means there is a duplicate of ClientIP.
####86 284.055938 VMware_##:##:## → Broadcast    ARP 60 ARP Announcement for ClientIP (duplicate use of ServerIP detected!)
// After duplicate IP was claimed from different machien by ARP Annoucement, Client didn't get any reponse from server.
###87 284.060408 ClientIP → ServerIP NFS 290 V4 Call LOCKU FH: 0xba250967 Offset: 0 Length: <End of File>
###88 284.064470 ClientIP → ServerIP NFS 290 V4 Call LOCKU FH: 0x814b29af Offset: 0 Length: <End of File>
###89 284.067039 ClientIP → ServerIP NFS 290 V4 Call LOCKU FH: 0xf797683f Offset: 0 Length: <End of File>
###90 284.069507 ClientIP → ServerIP NFS 290 V4 Call LOCKU FH: 0xccf948f7 Offset: 0 Length: <End of File>
###91 284.076393 ClientIP → ServerIP TCP 290 [TCP Retransmission] 834 → 2049 [PSH, ACK] Seq=23616618 Ack=18083401 Win=6568 Len=224 TSval=2533170399 TSecr=1803409694
###92 284.286480 ClientIP → ServerIP TCP 1158 [TCP Retransmission] 834 → 2049 [PSH, ACK] Seq=23615750 Ack=18083401 Win=6568 Len=1092 TSval=2533170609 TSecr=1803409694
###93 284.694446 ClientIP → ServerIP TCP 1158 [TCP Retransmission] 834 → 2049 [PSH, ACK] Seq=23615750 Ack=18083401 Win=6568 Len=1092 TSval=2533171017 TSecr=1803409694

Environment

VMware vSAN

Cause

This could happen because of duplicate use of Client IP. You might feel the vSANFS NFS server stopped responding, but in fact, vSANFS NFS was trying to send packet to a different machine who insisted to have same IP that NFS Client had. It's not a product but, but a basic network issue.

Resolution

If you have a packet capture, you're able to identify the mac address of a machine that insisted duplicate IP. Identify the machine based on the mac address and deal with the incorrect network configuration that caused duplicate IP.