Linux virtual machines with NFSv3 mounts experience an operating system hang after more than 15 minutes outage on the upstream datapath

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Applications interacting with an NFSv3 mount experience a hang on hard mounted NFSv3 mounts or an error on soft mounted NFSv3 mounts after more than 15 minute upstream data path failure
• The virtual machine performance increasingly degrades with the time it resides in an NFS hung state

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.0.x

Cause

This issue occurs when upstream connectivity issues last for longer than 15 minutes and the sunrpc/NFSv3 package attempts to reuse the original source and destination port of the previous connection without properly sending an RST on FIN packet.

Because of this, the TCP flow is not expired in the NSX Distributed Firewall (DFW), causing it to reject any new SYNs sent on the same source/destination port.

It is recommended to open a support request with your Linux vendor pertaining to the TCP behavior in the sunrpc/NFSv3 packages.

Resolution

This is a known issue affecting VMware NSX for vSphere 6.x.

Currently, there is no resolution.

To verify that you are experiencing this issue:

1. Run the netstat -n |grep 2049 command and look for any SYN_SENT lines on the virtual machine that is experiencing the issue.
   
   For example:
   
   netstat -n |grep 2049
   
   tcp 0 0 172.x.x.x:936 172.x.x.x:2049 SYN_SENT
   
   
2. On the ESXi host where the virtual machine resides, run this command to find the associated DFW filter name for the virtual machine/virtual NIC:
   
   # summarize-dvfilter | less
   
   You see output similar to:
   
   world 59441 vmm0:test1 vcUuid:'xx xx xx xx'
   port 67108909 test1.eth0
   vNic slot 2
   name: nic-59441-eth0-vmware-sfw.2
   agentName: vmware-sfw
   state: IOChain Attached
   vmState: Detached
   failurePolicy: failClosed
   slowPathID: none
   filter source: Dynamic Filter Creation
   
   
3. Dump the current flows for the filter, if an existing flow is in place for the same Source port / Destination port pair the issue is present:
   
   [root@host:~] vsipioctl vsipfwcli -f nic-59441-eth0-vmware-sfw.2 -c 'describe connection stat;'
   0x5608efe80000e0c3 af 2 ethertype 0x0800 proto tcp 172.xx.xx.xx:41510 -> 172.xx.xx.xx:80 e6 24a 5 3
   0x5608efe80000e0c4 af 2 ethertype 0x0800 proto tcp 172.xx.xx.xx:41512 -> 172.xx.xx.xx:80 e6 24a 5 3
   0x5608efe80000e0c5 af 2 ethertype 0x0800 proto tcp 172.xx.xx.xx:41518 -> 172.xx.xx.xx:80 e6 29a 5 5
   0x5608efe80000e0c6 af 2 ethertype 0x0800 proto tcp 172.xx.xx.xx:936 -> 172.xx.xx.xx:2049 55e8 44288 92 d4
   0x5608efe80000e0c7 af 2 ethertype 0x0800 proto tcp 172.xx.xx.xx:41520 -> 172.xx.xx.xx:80 10e 272 6 4

To work around this issue, use one of these options:

• Add the affected virtual machine to the NSX Manager exclusion list, and then remove it from the exclusion list. This removes all currently tracked flows and re-applies the Firewall. For more information, see the Exclude Virtual Machines from Firewall Protection section in the NSX Administration Guide.

• Add all Linux virtual machines utilizing NFS to the Exclusion list, or disable the NSX Distributed Firewall in the Cluster where these virtual machines reside.

• Implement NFSv4, which properly sends a RST when tearing down the established connection prior to moving to SYN_SENT state.

Note: The preceding link was correct as of October 27, 2015. If you find the link is broken, please provide feedback and a VMware employee will update the link.

 

Additional Information

This is an issue that pertains to the TCP behavior of the Linux sunrpc / NFSv3 packages.