Linux virtual machines using EFI firmware may fail to reboot on ESXi 5.5

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When a Linux virtual machine configured to use EFI firmware is rebooted, the operating system may appear to hang during boot. This is more likely to occur upon a third or subsequent reboot of a virtual machine, and is more likely to occur if the virtual machine is configured with at least several gigabytes of RAM. This problem has been noted to occur with recent patch releases of Red Hat Enterprise Linux 6.x and 7.x.
• If the virtual machine is configured to log to a console (for example, by adding a serial port and configuring the guest kernel to log to the serial port), a message similar to the following will appear in the console log:

    WARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing ____MB of RAM and the guest OS will panic with a trace similar to the following:
 

    init_memory_mapping: 0000000000000000-00000000000a0000
    0000000000 - 00000a0000 page 4k
    kernel direct mapping tables up to a0000 @ 8000-b000
    PANIC: early exception 0e rip 10:ffffffff8152e5fa error 0 cr2 ffff880001a8fff0
    Pid: 0, comm: swapper Not tainted 2.6.32-642.el6.x86_64 #1
    Call Trace:
    [<ffffffff81046069>] ? native_read_cr2+0x9/0x10
    [<ffffffff81c3a19e>] ? early_idt_handler+0x5e/0x71
    [<ffffffff8152e5fa>] ? init_memory_mapping+0x3ba/0x6a0
    [<ffffffff8152e5cb>] ? init_memory_mapping+0x38b/0x6a0
    [<ffffffff81c407c0>] ? setup_arch+0x5e7/0xccb
    [<ffffffff8107db7e>] ? vprintk_default+0xe/0x10
    [<ffffffff81546f02>] ? printk+0x4f/0x55
    [<ffffffff81c3adda>] ? start_kernel+0xdc/0x431
    [<ffffffff81c3a33a>] ? x86_64_start_reservations+0x125/0x129
    [<ffffffff81c3a453>] ? x86_64_start_kernel+0x115/0x124
    RIP init_memory_mapping+0x3ba/0x6a0

Environment

VMware vSphere ESXi 5.5

Cause

An issue with the way ESXi 5.5's EFI firmware configures the virtual CPU's Memory Type Range Registers (MTRRs) during restart may cause some guest operating systems to mark most of the virtual machine's memory as unusable. The guest OS then panics due to an out-of-memory condition.

Resolution

A virtual machine which has failed to reboot may be recovered by powering it off and then powering it back on again; During a cold boot, the guest operating system should start normally.
An affected virtual machine may once again fail during a third or subsequent reboot unless the problem is mitigated using the following instructions:

• The failure may be avoided by adding the "disable_mtrr_trim" parameter to the guest kernel command-line. Consult your guest operating system documentation for instructions on configuring the bootloader with custom kernel parameters.
• The failure may also be avoided by substituting a reboot with a cold restart: If the virtual machine is a potentially affected configuration and a guest OS reboot is required, shut down the virtual machine instead, and then power it back on again.