vSAN File Services -- General Recommendations in regards to evaluating/improving Performance

Products

• VMware vSAN
• VMware vSAN 8.x

Issue/Introduction

General overview on available Public information to assist with investigation of Performance concerns when using vSAN File Services

in cases where the vSAN Healthcheck is not reporting any issues and where changes on the physical Network Layer did not result in improvements.

 

Example Performance concern:

• How to improve VSFS file transfer performance

 

Examples of Changes made which did not improve vSAN File Services Performance:

• Usage of vDS
• Implementation of MTU 9000 (Jumbo Frame) across vDS, vSAN File Services vmk and vmnics
• Using of 10 Gbps Uplinks

 

Environment

vSAN 8.x

vSAN 9.x

Cause

A.) vSAN File Services operates by deploying a File Service VM (FSVM) on each host in the vSAN Cluster. Performance limits are typically reached due to the following factors:

• FSVM Resource Constraints: Each FSVM is a virtual appliance with fixed CPU and RAM. If a single share (hosted by a specific FSVM) is overloaded, that FSVM's compute resources may bottleneck
• Single-Share Affinity: A single vSAN FS share is served by one FSVM at a time. Therefore, the performance of a single share is limited by the power of a single node, regardless of cluster size
• Protocol Overhead: NFS and SMB protocols introduce additional metadata overhead compared to raw block storage (vDisk), especially for "small file" workloads or high-frequency metadata operations (e.g., ls, getattr)

 

B.) Issues on the physical Network Layer can degrade Performance, such as:

• Bandwidth constrains/issues
• Configuration of the dedicated VMkernel Ports
• HW related issues with Cabling, SFP, vmnics etc. (= involved Network cards)
• Physical Switch issues

 

C.) Review the Storage Policy applied to vSAN File Services 

Since File shares live as vSAN objects, with resilience defined by the storage policy associated with the share.

• RAID 1 mirroring requires fewer I/O operations to the storage devices, so it can provide better performance. 
• Change high-performance VMs to RAID-1 (Mirroring) for better speed.

Resolution

1.) Review important information to understand vSAN File Services Architecture and Best Practices of implementation:

• Checklist Summary for vSAN Network (which also affects vSAN File Services)
• Networking Considerations for vSAN File Service
• Limitations and Considerations of vSAN File Service
• Provides some Background information in regards to vSAN File Services under 8.x for OSA Clusters

• Scale-Out via Multiple Shares
  Because a single share is pinned to a single FSVM, consider increasing aggregate Cluster performance by distributing your data across multiple shares.
  Recommendation: Instead of one 5TB share, create five 1TB shares. This allows vSAN to distribute the load across multiple FSVMs and multiple hosts.

 

2.) Start Monitoring vSAN File Services Performance

• Review the Performance Graphs in the vSphere Client: Version 8.x, Version 9.x
• Monitor FSVM Resource Usage
  Check if the FSVM is CPU or Memory bound during peak I/O.
  by using esxtop or vCenter Performance Charts to monitor the FSVM appliances (= FSVM > Tab Monitor > Performance > Advanced )
  If CPU usage is consistently near 100%, consider offloading workloads to additional shares.
• Monitor NFS client IO profiles to ensure VSAN object congestion is not occurring via esxtop ( Guide - Section 4 Disk )

 

If assistance is required, open a Ticket with VMware by Broadcom Support.

 

Additional Information

• Overview of vSAN File Services in VMware Cloud Foundation 9.1 - Page 13 - Zero-Copy Data Path for Optimal Performance 
• Enhancements with vSAN File Services in VMware Cloud Foundation 9.0