Understanding the VMFS heap eviction process in ESXi 5.5 and later

Products

• VMware vSphere ESXi

Issue/Introduction

This article explains how the heap eviction process improved in ESXi 5.5 and later.

For earlier versions of ESXi, see ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage (1004424).

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0

Resolution

To address an issue with VMFS heap while accessing 30 TB of open files from a single ESXi host, ESXi 5.0 Patch 05 and ESXi 5.1 Update 1 introduced a larger heap size.

VMFS5 double-indirect pointer blocks were introduced to accommodate very large VMDKs built with 1 MB file blocks. Now the file descriptor pointed to a pointer block, the entries in this pointer block pointed to another pointer block, and this contained entries that pointed to the data block. These pointer blocks were saved in an area called the PB Cache which was allocated from the VMFS heap, which resulted in issues like heap depletion and some undesired side effects.

From ESXi 5.5 onwards, pointer block caching does not use the same heap as VMFS3 heap. Instead they are allocated separately. This sizing is controlled by /VMFS3/MinAddressableSpaceTB and /VMFS3/MaxAddressableSpaceTB values.

The size of the VMFS heap is now irrelevant with respect to how much open addressable space there can be.

Default, maximum and minimum heap amount table for ESXi 5.5 and later.

 

Maximum and Default heap amount	256 MB
Minimum heap amount	16 MB

MaxAddressableSpaceTB defines how many pointer blocks for open files are cached in memory before the eviction mechanism starts. The default value is set to 32 TB, but this can be set up to a maximum value of 128 TB.

The default open VMDK storage per host is 32 TB and it can be increased up to 128 TB. But performance varies depending on the working set (active pointer blocks required). The MaxAddressableSpaceTB controls the growth of PB cache. After the cache reaches 80% usage, eviction starts and space is reused. If customer has a working set beyond 128 TB (that is the max PB cache can handle), performance may not be optimal.

 

To change the value of /VMFS3/MaxAddressableSpaceTB to 128 TB using the ESXi console:

 

# esxcli system settings advanced set -i 128 -o /VMFS3/MaxAddressableSpaceTB

 

To confirm that MaxAddressableSpaceTB value is set correctly, run this command:

# esxcli system settings advanced list -o /VMFS3/MaxAddressableSpaceTB

 

The value of the advanced option /VMFS3/ MaxAddressableSpaceTB can also be changed using the vSphere Web Client, the vSphere Client, or using PowerCLI. For more information, see Configuring advanced options for ESXi/ESX (1038578).

Additional Information

Alternative commands to set and check the value of MaxAddressableSpaceTB:

 

To set the value:

 

# esxcfg-advcfg -s 128 /VMFS3/MaxAddressableSpaceTB
 

To check the value:

 

# esxcfg-advcfg -g /VMFS3/MaxAddressableSpaceTB

 

 

To get pointer block cache (pbcache) usage statistics, run this command:

esxcli storage vmfs pbcache get

For example:

esxcli storage vmfs pbcache get

 

Note: This command only works on 6.0

Cache Capacity Miss Ratio: 0 %
Cache Size: 64 MiB
Cache Size Max: 132 MiB
Cache Usage: 49 %
Cache Working Set: 15 TiB
Cache Working Set Max: 32 TiB
Vmfs Heap Overhead: 238 KiB
Vmfs Heap Size: 24 MiB
Vmfs Heap Size Max: 256 MiB

 

ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage
Configuring advanced options for ESXi/ESX
ESXi 5.5 における VMFS ヒープの扱いについて