Experiencing latency issues on NFS file shares.

Products

• VMware vSAN

Issue/Introduction

Symptoms : 

• Uploading a file to a VM residing on the NFS file share is faster.
• Uploading many directories (containing many files) to a VM residing on the NFS file share is slower compared to uploading a single file of same size.
• Initial upload might be faster for few GBs. But slows down gradually.

Environment

VMware vSAN 6.x
VMware vSAN 7.x
VMware vSAN 8.x

Cause

Transferring a single large file over an NFS file share is generally faster than transferring multiple smaller files of the same total size. This is due to the overhead associated with file system operations and network communication for each individual file.

Here's a breakdown of the reasons:

• Reduced Metadata Overhead: Each file requires metadata (like directory entries, file attributes) to be created and managed by the file system. With a single large file, this metadata overhead is incurred only once, whereas with multiple smaller files, it's incurred for each file, adding up significantly.
• Network Latency: Network file transfers involve round-trip communication between the client and the server for each file operation (open, read, write, close). This latency is multiplied for each file when transferring many smaller files, leading to a slower overall transfer speed. 
• File System Operations: Operating systems and file systems have overhead associated with opening, reading, writing, and closing files. This overhead is minimized when transferring a single large file but becomes a significant factor when dealing with numerous smaller files.

The initial faster upload speed on an NFS share, followed by a slowdown, is likely due to NFS protocol limitations and the way it handles large file transfers. Specifically, NFS often uses smaller packets for initial data transfer, which can lead to a faster start, but as the file size increases, file transfer become slowdown.

Resolution

This is an expected behavior.