ESXi memory capacity might be less than the sum of DIMMs installed due to advanced memory configuration in server BIOS/Firmware

Products

• VMware vSphere ESXi

Issue/Introduction

A server running VMware vSphere ESXi 7.0.x shows less overall memory in both vSphere Client and ESXi host client, than is physically installed.

For example, the server has 24 DIMMs installed on the mainboard, each with a capacity of 32 GByte, aka 768 GByte overall. However, the ESXi hypervisor software only sees 384 GByte memory.

Environment

VMware vSphere ESXi 7.0.x

Cause

Many server vendors offer an option to enable configure advanced memory configurations for increased failure tolerance, such as:

• memory mirroring, full and partial
• spare mode

These features work by reserving memory modules on a hardware BIOS level, thus resulting in those modules no longer being available to the operating system. For example, a server has 24 x 32 GByte memory installed, so without any other options, the OS will see  768 GByte. However, if full mirroring is enabled in BIOS, half of these modules will be marked as reserved, and only the second half of them can be used by the OS, so the latter will only see 384 GByte.

Similarly any module marked as spare will be no longer accessible to the operating system, leading to the available memory size being reduced by the number of spare DIMMs. 

Resolution

To see which memory modules are available to the ESXi software, you can use the command to query the IPMI:

# localcli hardware ipmi sdr list -p -i -n all

The output of the command will look like in the following example:

Node-Sensor  Description                                       Entity-Instance  Computed Reading       Base Unit    Raw Reading  Sensor Type       Timestamp/Comment  
-----------  ------------------------------------------------  ---------------  ---------------------  -----------  -----------  ----------------  -------------------
0.3          Other 1 01-Inlet Ambient                          64.1             18                     degrees C    18           Temperature       YYYY-MM-DDTHH:MM:SS
0.4          Other 1 02-CPU 1                                  65.1             40                     degrees C    40           Temperature       YYYY-MM-DDTHH:MM:SS
0.5          Other 2 03-CPU 2                                  65.2             40                     degrees C    40           Temperature       YYYY-MM-DDTHH:MM:SS
0.7          Memory Device 2 05-P1 DIMM 7-12                   32.2             24                     degrees C    24           Temperature       YYYY-MM-DDTHH:MM:SS
0.9          Memory Device 4 07-P2 DIMM 7-12                   32.4             24                     degrees C    24           Temperature       YYYY-MM-DDTHH:MM:SS

 

In the output, look for the memory devices and compare them with the number physically installed in the server. If the amount listed by IPMI is less than the count of physical modules, please reach out to your hardware vendor.

 

Additional Information

These are some hardware references from the respective hardware vendors:

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001068en_us&page=GUID-F8C8A028-EB57-4EAC-A664-331EAC4EA821.html

https://www.dell.com/support/manuals/en-sg/poweredge-r730xd/r730xd_ompublication/mode-specific-guidelines?guid=guid-30abb8bf-8dd5-4139-b67f-82bc5c5be1f5&lang=en-us

https://pubs.lenovo.com/sr950/memory_module_installation_order_sparing