VMware vSAN Support for large cache tier capacity (>600GB, up to 1.6TB) in vSAN 8.0

Products

• VMware vSAN

Issue/Introduction

The vSAN cache tier has historically be limited to 600GB in vSAN 7.0 and earlier.

In vSAN 8.0x we now support higher cache tier capacities up to 1.6TB.

 

In order to take advantage of additional cache it requires changing the parameter of each ESXi hosts advanced settings.

 

The following information outlines key benefits and caveats when enabling.

 

Impact/Risks:
Note: Support for utilizing larger cache disks is only supported on All-Flash configurations! Hybrid clusters are not supported.

Caveats of large cache tier support:

1. Increased memory requirement - Every vSAN disk group with large cache tier will incur an additional memory cost of 5GB. With the max number of supported disk groups per host (which is 5), the max additional memory cost increase is 25GB per host.
2. The full benefits of large cache tier might not be realized if all disk groups in a cluster are not large cache tier enabled.

Benefits of large cache tier support:

1. Virtual applications  that are write intensive or perform large writes will take advantage of this optimization.
2. Customers with existing cache devices larger then 600 GB are now able to now leverage this advancement in vSAN technology.
3. Increasing cache size improves sequential write and larger block random write workload cases with burst phase duration lasting 2x-3x more than in 7.0 U3 (with a cache tier of 600GB only).
4. Small Working Set Size (WSS) cases also provide higher benefit (up to 10x improvement) for larger block writes in IOPs, Latency, and CPU efficiency.
5. Regular reboot duration significantly improved (up to 50%) because higher map entries can be cached with a large cache tier.
6. Resync duration improved significantly (up to 0.5x) with Rebuild/EMM type workloads.

Environment

VMware vSAN 8.0.x

Cause

The support for large cache tier has been a demand from our customers for some time now. The flash disk capacities are increasing and vSAN was only being able to use up to 600GB. This limit has been increased to benefit from larger cache disks.

Resolution

Note: Support for utilizing larger cache disks is only supported on All-Flash configurations! Hybrid clusters are not supported.

To enable large cache tier, the change can be performed either from the vCenter UI or the Host UI via Advanced settings, or alternatively via CLI.

Via vSphere UI
vCenter - Cluster - Click on the host - Configure tab - Advanced System Settings

Via ESXi Host Client
On the ESXi Host Client: Host - Manage - System - Advanced Settings.


Via CLI
From CLI the change can be performed by running the below command:
esxcfg-advcfg -s 1 /LSOM/enableLargeWb

To check the current setting run:
esxcfg-advcfg -g /LSOM/enableLargeWb

A value of 0 means it's disabled and a value of 1 means it's enabled

Notes:

• To enable large cache tier support vCenter and ESXi must be on version 8.0 or higher and on-disk format must be version 17 or higher
• Large cache tier is only supported on new clusters or entire clusters getting fully upgraded to the above versions.
• For existing disk groups, changing the advanced setting must be followed by Managing Disk Groups and Devices
• There is no reboot required after changing this setting.

Additional Information