To address an issue with VMFS heap while accessing 30 TB of open files from a single ESXi host, ESXi 5.0 Patch 05 and ESXi 5.1 Update 1 introduced a larger heap size.
VMFS5 double-indirect pointer blocks were introduced to accommodate very large VMDKs built with 1 MB file blocks. Now the file descriptor pointed to a pointer block, the entries in this pointer block pointed to another pointer block, and this contained entries that pointed to the data block. These pointer blocks were saved in an area called the PB Cache which was allocated from the VMFS heap, which resulted in issues like heap depletion and some undesired side effects.
From ESXi 5.5 onwards, pointer block caching does not use the same heap as VMFS3 heap. Instead they are allocated separately. This sizing is controlled by /VMFS3/MinAddressableSpaceTB and /VMFS3/MaxAddressableSpaceTB values.
The size of the VMFS heap is now irrelevant with respect to how much open addressable space there can be.
Default, maximum and minimum heap amount table for ESXi 5.5 and later.
Maximum and Default heap amount | 256 MB |
Minimum heap amount | 16 MB |
MaxAddressableSpaceTB defines how many pointer blocks for open files are cached in memory before the eviction mechanism starts. The default value is set to 32 TB, but this can be set up to a maximum value of 128 TB.
The default open VMDK storage per host is 32 TB and it can be increased up to 128 TB. But performance varies depending on the working set (active pointer blocks required). The MaxAddressableSpaceTB controls the growth of PB cache. After the cache reaches 80% usage, eviction starts and space is reused. If customer has a working set beyond 128 TB (that is the max PB cache can handle), performance may not be optimal.
To change the value of /VMFS3/MaxAddressableSpaceTB to 128 TB using the ESXi console:
# esxcli system settings advanced set -i 128 -o /VMFS3/MaxAddressableSpaceTB
To confirm that MaxAddressableSpaceTB value is set correctly, run this command:
# esxcli system settings advanced list -o /VMFS3/MaxAddressableSpaceTB