Support for large cache tier capacity (>600GB, up to 1.6TB) in vSAN 8.0

Products

• VMware vSAN

Issue/Introduction

The vSAN cache tier is currently capped at 600GB in vSAN 7.0 and earlier.

Starting with vSphere 8.0, vSAN supports higher cache tier capacities, up to 1.6TB. However, this is not enabled by default. By default, any new disk groups getting created will still use cache tier capacity of only up to the existing limit of 600GB.

There are benefits and caveats to enabling this support for large cache tier capacity. The KB is to call these out so that informed decision whether or not to enable large tier capacity can be made and how to enable it.

 

Impact/Risks:
Note: Support for utilizing larger cache disks is only supported on All-Flash configurations! Hybrid clusters are not supported.

Caveats of large cache tier support:

1. Increased memory requirement - Every vSAN disk group with large cache tier will incur an additional memory cost of 5GB. With the max number of supported disk groups per host (which is 5), the max additional memory cost increase is 25GB per host.
2. The full benefits of large cache tier might not be realized if all disk groups in a cluster are not large cache tier enabled.

Benefits of large cache tier support:

1. Workloads that are data heavy or workloads with bottlenecks at cache size will benefit from a larger cache tier.
2. Customers can better utilize their larger capacity SSDs towards the vSAN cache tier than be limited to 600GB.
3. Increasing cache size improves sequential write and larger block random write workload cases with burst phase duration lasting 2x-3x more than in 7.0 U3 (with a cache tier of 600GB only).
4. Small Working Set Size (WSS) cases also provide higher benefit (up to 10x improvement) for larger block writes in IOPs, Latency, and CPU efficiency.
5. Regular reboot duration significantly improved (up to 50%) because higher map entries can be cached with a large cache tier.
6. Resync duration improved significantly (up to 0.5x) with Rebuild/EMM type workloads.

Environment

VMware vSAN 8.0.x

Cause

The support for large cache tier has been a demand from our customers for some time now. The flash disk capacities are increasing and vSAN was only being able to use up to 600GB. This limit has been increased to benefit from larger cache disks.

Resolution

Note: Support for utilizing larger cache disks is only supported on All-Flash configurations! Hybrid clusters are not supported.

To enable large cache tier, the change can be performed either from the vCenter UI or the Host UI via Advanced settings, or alternatively via CLI.

Via vSphere UI
vCenter - Cluster - Click on the host - Configure tab - Advanced System Settings

Via ESXi Host Client
On the ESXi Host Client: Host - Manage - System - Advanced Settings.


Via CLI
From CLI the change can be performed by running the below command:
esxcfg-advcfg -s 1 /LSOM/enableLargeWb

To check the current setting run:
esxcfg-advcfg -g /LSOM/enableLargeWb

A value of 0 means it's disabled and a value of 1 means it's enabled

Notes:

• To enable large cache tier support vCenter and ESXi must be on version 8.0 or higher and on-disk format must be version 17 or higher
• Large cache tier is only supported on new clusters or entire clusters getting fully upgraded to the above versions.
• For existing disk groups, changing the advanced setting must be followed by recreating the disk group.
• There is no reboot required after changing this setting.

Additional Information

Increased Write Buffer Capacity for the vSAN 8 Original Storage Architecture