An ESX host fails to boot after increasing the Service Console or system memory
search cancel

An ESX host fails to boot after increasing the Service Console or system memory

book

Article ID: 337166

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction


Symptoms:
After increasing the amount of physical RAM or Service Console memory, you may experience these symptoms:
  • An ESX fails to boot
  • You may see a purple diagnostics screen that contains an error similar to:

    NOT_IMPLEMENTED /build/mts/release/bora-84374/bora/vmkernel/main/util.c:83


Environment

VMware ESX 4.0.x
VMware ESX Server 3.5.x

Resolution

Cause

This issue occurs when the ESX VMkernel cannot allocate sufficient physical memory below 4GB during boot.
The ESX host requires memory below 4GB (called Low Memory) for the Service Console, drivers, and other uses. The Service Console is configured to use between 272-800MB of the lowest physical memory. The ESX VMkernel requires some low memory to keep trace of high memory pages. The exact amount of memory Reserved can be seen with cat /proc/vmware/mem.
Example Reservations are as follows:
  • With 256GB of physical memory installed in a server, the ESX VMkernel reserves 1289MB of Low Memory.
  • With 128GB of physical memory installed in a server, the ESX VMkernel reserves 644MB of Low Memory.

The relationship between Physical Low and High Memory is as follows:

  • Maximum Console OS memory = Total Low Memory - VMkernel Reserved Low Memory - ~100MB Drivers
The total amount of low memory available is typically 4GB. However, the System BIOS can reserve some of this memory, so less than 4GB is available to the operating system. At a high level, the memory map looks like the figures below:
The following lines within the /var/log/messages log file represent the memory map as presented by the BIOS:
Note: Memory ranges from 000000007ff76300 (2047MB) to 0000000100000000 (4096MB) are reserved by the BIOS and are not usable by the OS.

The following lines from /var/log/messages record the VMkernel interpretations of the RAM map:

Note: The line containing linux MA reflect the Machine Address memory ranges allocated (in bytes) to the service console. The Service Console is using the lower 272MB. The lines containing vmkernel MPN reflect the Machine Page Numbers (4k blocks) allocated for the VMkernel. The VMkernel is using memory ranges 272MB - 2047MB and 4GB - 256GB.

The table below contains several combinations of Physical Memory and Service Console Memory assignment, along with their impact on the amount Unreserved Low Memory available when 2GB of low memory is reserved by the BIOS. The combination of 256GB of Physical Memory and 800MB of Service Console Memory with only 2GB of Low Memory leaves the server with only 7MB of Unreserved Low Memory, insufficient for booting. In this case the ESX VMkernel startup stops with the purple diagnostics screen.

Impact of Memory Allocations
Physical MemoryPhysical Low MemoryReserved Low MemoryService Console MemoryUnreserved Low Memory
256 GB4096 MB1289 MB272 MB2535 MB
256 GB4096 MB1289 MB800 MB2007 MB
256 GB2048 MB1289 MB272 MB487 MB
256 GB2048 MB1289 MB800 MB7 MB
128 GB4096 MB644 MB272 MB3180 MB
128 GB4096 MB644 MB800 MB2652 MB
128 GB2048 MB644 MB272 MB1132 MB
128 GB2048 MB644 MB800 MB604 MB

Resolution

For servers affected by this limitation, it is recommended that the Service Console memory be reduced to between 500MB - 650MB, or that the Physical System Memory be reduced by 16GB - 32GB. Make one of the following changes to increase the amount of Low Memory available below 4GB:
  • Reduce the amount of Physical Memory in the server.
  • Reduce the amount of Physical Memory reserved by the BIOS.

    Note: The BIOS memory reservation may or may not be configurable in a particular server. Consult your hardware vendor for more information.
  • Reduce the amount of Service Console Memory assigned by using the following steps:
    1. Restart the ESX host.
    2. When presented the boot menu, ensure you have the default boot option highlighted.
    3. Press the e key to edit the default boot option. A new list appears.
    4. Use the arrow keys to highlight the line which starts with kernel /vmlinuz.
    5. Press the e key to edit the line in the boot configuration. A prompt with the entire line will appear.
    6. Use the arrow key to move over the portion of the line which has a parameter similar to the following:

      mem=800M
    7. Change the value to something between 500 and 650 megabytes. The format is mem=M.

      Note: The exact threshold may vary depending on the hardware configuration. To determine the exact value, engage your hardware vendor or test different values.
    8. Press enter to accept the changes.
    9. Press the b key to boot the host using the configuration.

      Note: The configuration change is not permanent. To make the change permanent, proceed to step 10.
    10. Wait for the host to boot.
    11. When the host has booted, open the vSphere Client and connect to the host.
    12. Under the configuration tab, change the memory properties of the ESX host to ensure that the RAM assignment has been reduced. For more information on adjusting the RAM assignment using the vSphere Client, see Increasing the amount of RAM assigned to the ESX Server Service Console (1003501).