Large data transfer to a new NFS4.1 datastore causes the VM to lock up or stop responding

Products

• VMware vSphere ESXi

Issue/Introduction

When doing a large write in to a VM on an NFS4.1 datastore running from a NetApp storage array with code version 9.16 the VM will lock up or stop responding but no errors will be seen in vCenter. 
This is normally but not limited to an in guest OS operation .
This can be from a basic file copy, a restore operation, or migration. 

In the vmkernel logs you can see messages related to SunRPC like the following:

SunRPC: 5704: Socket space full. rpc queued.

Environment

ESXi (All) 
NFS4.1 
NettApp code 9.16

Cause

This is due to the Storage array failing to reply to an RPC CALL - this is a documented behavior from NetAPP 9.16 versions (see additional information section) 

Resolution

Review with storage array vendor (NetAPP) to confirm and implement fix. 

If additional assistance is required, open a case with Broadcom support for a review of the interaction between ESXi and the storage array 

Additional Information

CONTAP-484943: Some NFS ops against qtrees taking 5 minutes on 9.16.1

NFS Datastore File Locking and Latency Issues on ESXi Hosts with Storage I/O Control Enabled