Support for large cache tier disks in a Hybrid Disk Group

Products

• VMware vSAN

Issue/Introduction

Cache tier disks in a hybrid disk group has a max Write buffer of 600GB and the rest of the cache disk is used as read cache. 

So a 2TB cache tier disk breaks down like so:
Cache Tier Disk Size = 2 TB
Write Buffer Size = 600 GB
Read Cache Size = 1.4 TB

Environment

VMware vSAN OSA (ALL Versions)

Hybrid Disk Groups

Resolution

• To get the best read performance of these larger cache tier disks we need to increase the LSOM Heap size.
• LSOM Heap Size default is 256MB.
• /LSOM/heapSize is global, so we need to find the total memory footprint required for all the disk groups of the host and if it is more than the default(256MB), then we need to set it accordingly. To do this follow the steps below.

1) Calculate the required heap size per disk group by using the below formula:

Disk_Group_Heap_Size = Heap_size_read_cache_per_DG  = (read_cache_size >> 20) * 34 bytes
read_cache_size >> 20 => This means how many MB of data in Read Cache(Not whole Cache Tier Disk) as 34 Bytes per MB is reserved.
Using a 3TB disk for example we have:

Cache Tier Disk Size = 3 TB
Write Buffer Size = 600 GB
Read Cache Size = 2.4 TB

Memory foot print will be calculated for only Read Cache size(2.4 TB). 
2.4 TB is larger than 20MB so 2.4 * 34 Bytes = 81.6MB round to the nearest whole number we get 82MB

2) Now that we have the LSOM Heap Size for individual disk group we need to calculate the total heap size if the host has more than one disk group with the below formula:

Total_Heap_Size =  Heap_size_read_cache_DG1 + Heap_size_read_cache_DG2 + ect.
Example: host with 4 disk groups: 82*4=328, 328 > 256 so we need to increase the LSOM/heapSize

3) Run esxcfg-advcfg -g /LSOM/heapSize to get the current heap size

4) Run esxcfg-advcfg -s Total_Heap_Size /LSOM/heapSize to set the heap size you just calculated if it's more than 256MB.
Example: esxcfg-advcfg -s 328 /LSOM/heapSize

5) Reboot the host so the new setting takes affect

Note: This is per host so you will need to do this for all hosts in the cluster.

The max LSOM Heap Size is 2048 MB which means the max cache tier disk per disk group is 12TB
Memory required for 1TB read cache size: 34 MB

Max Read cache size supported = Total Read Cache size
                         =  (2048/34) TB 
                         = 60.23 TB

Max DGs per Host = 5
Max write Buffer Size per DG = 600 GB
Total Write buffer Size per host = 600 GB * 5 = 3 TB.

Max Read Cache Size per Host = 60.23 TB 
Max Cache Disk Size per host = 60.23 TB + 3 TB = 63.23 TB ≈ 63 TB

If we divide this evenly across 5 DG.

Max Read Cache size per DG = 12.04 TB
Max Cache Disk Size per DG = 12.04 TB + 600 GB = 12.64 TB ≈ 12 TB