ESXi hosts with larger workloads may generate partial core dumps
search cancel

ESXi hosts with larger workloads may generate partial core dumps

book

Article ID: 310560

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • The ESXi 5.x host becomes unresponsive.
  • The host displays a purple diagnostic screen.
  • You see the backtrace:

    Coredump to disk. Slot 1 of 1. DiskDump: Partial Dump: Out of space ....

  • Attempts to extract the core dump file fail.
  • The core dump file is truncated.
  • When extracting the core dump, you see errors similar to:

    • Warning: One or more coredump offsets is zero. You may have a partial dump. Examine header (using '-h') for details.
    • Offset check failed for 'Code'. Header expected stream offset 0, but actual stream offset is 75293512.
      Coredump extraction was not fully successful. Corefile is incomplete but may still be usable.
    • Partial coredump detected. Most parts of the coredump are probably usable,but sparse regions may not be present.


Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.0

Cause

This issue occurs because the default slot size for the core dump partition cannot accommodate a complete core dump of an ESXi host that is under heavy load.

Resolution

The default partitioning scheme and VMkernel core partition for ESXi cannot be modified to accommodate for larger core dumps. To be able to record core dumps that are larger than the default VMkernel core partition, an alternative location must be defined. This alternative location can be:

  • An ESXi Dump Collector service
  • A larger Diagnostic partition located on a USB device
  • A larger Diagnostic partition located on logical unit or alternative local disk

The sections below provide guidelines on implementing these alternative options.

Configure your ESXi hosts to use the ESXi Dump Collector service

To configure your ESXi hosts:

  1. Install and configure the vSphere ESXi Dump Collector service. For more information, see the Install vSphere ESXi Dump Collector section in the vSphere Installation and Setup Guide.
  2. Configure your host to direct core dumps over the network. For more information, see the Managing Core Dumps section in the vSphere Command-Line Interface Documentation Guide.

When using an ESXi Dump Collector service for the purposes of collecting VMkernel core dumps, configuring a larger slot size is not required. However, VMware recommends a minimum of 300 MB of free space on the ESXi dump collector core dump location to ensure that the resulting core dump is not truncated.

Configure your ESXi hosts to use a local USB device

Add a USB storage device that has a capacity greater than 300 MB. To configure your host to use a local USB device for core dumps, see Configuring a vSphere ESXi host to use a local USB device for VMkernel coredumps (broadcom.com) 

Once the host is configured to use a USB device for a core dump, you must configure the ESXi host to use a core dump slot size as described in the ESXi hosts with larger workloads may generate partial core dumps (broadcom.com) section below.

Configuring your ESXi host to use a local or remote block device

Add a block storage device or provision a logical unit on the ESXi host that is greater than 300 MB. Use the add storage wizard to create a new diagnostic partition on the available storage device. For more information on creating a core dump partition, see the Creating a Diagnostic Partition section in the ESXi and vCenter Server 5 Documentation.

After creating the diagnostic partition, configure the ESXi host core dump slot size as described in the ESXi hosts with larger workloads may generate partial core dumps (broadcom.com)  section below.

Notes:

  • When installing ESXi, a VMkcore partition is created that is 110 MB in size. This partition size cannot be modified during or after the installation of ESXi.
  • Configuring a remote device using the ESXi host software iSCSI initiator is not supported. Any attempt to configure a remote device using the ESXi host software iSCSI initiator results with this error: Unsupported disk type: Software iSCSI LUNs are not supported

Configure your ESXi hosts to use a larger slot size

When installing ESXi, the default slot size for a core partition is 100 MB. The slot size can be modified on ESXi on these releases and later:

  • ESXi 5.0 Update 2
  • ESXi 5.0 Patch 3
  • ESXi 5.1

Note: Starting with ESXi 5.5, the diskDumpSlotSize equals automatically sized, otherwise requested size >=100 MB.

To modify the slot size so that the core dumps are not truncated, you must modify the diskDumpSlotSize to the desired size in megabytes. VMware recommends a value of 300 MB. Modifying the slot size can be done during boot, on the command line, or using the vSphere Client.

To set the diskDumpSlotSize during boot:

  1. Restart the ESXi system.
  2. When prompted, press Shift+O.
  3. At the prompt, add this entry (space delimited):

    diskDumpSlotSize=size_in_megabytes

  4. Press Enter and proceed with the boot process.
  5. Verify that the setting has been applied by running the command:

    esxcli system settings kernel list -o diskDumpSlotSize

  6. Confirm that the value under the Runtime column matches that of the value specified during boot.

For more information on setting advanced options, see the Enter Boot Options to Start an Installation or Upgrade Script section in the VMware vSphere ESXi and vCenter Server 5 Documentation.

To set the diskDumpSlotSize from the command line:

  1. Log in to the ESXi Shell environment or an environment with vCLI.
  2. Run the command:

    esxcli system settings kernel set --setting=diskDumpSlotSize --value=size_in_megabytes

  3. Verify that the setting has been applied by running the command:

    esxcli system settings kernel list -o=diskDumpSlotSize

To modify the diskDumpSlotSize value from the vSphere Client:

  1. Select the ESXi host from the Hosts and Clusters view in vCenter Server.
  2. Click the Configuration tab.
  3. Click Advanced Settings under Software.
  4. Under Vmkernel and then boot, locate the parameter called VMKernel.Boot.diskDumpSlotSize.
  5. Change this value to 300MB and then click OK.
    NOTE: This steps assumes that a diagnostic partition equal to or greater than 300 MB exists. Performing this step without a large enough diagnostic partition results in truncated core dump files as described in the symptoms section.

To modify the diskDumpSlotSize value from the vSphere Web Client in ESXi 5.1 and later.

  1. Select the ESXi host.
  2. Click on the Manage Tab.
  3. Select Advanced System Settings.
  4. In the Filter option on the right, type in SlotSize to search for the correct parameter.
  5. The option VMkernel.Boot.diskDumpSlotSize will be listed.
  6. Select the option and click on Edit on the left of the Advanced System Settings page.
  7. Change the slot size to size_in_megabytesMB and click OK.

For more information on setting advanced options, see Configuring advanced options for ESXi/ESX (broadcom.com) 

Additional Information

The slot size and VMkcore partition size are not synonymous. The slot size in ESXi 5.0 and later is configurable. The diskDumpSlotSize option must be set during the boot process of the initial installation. Do this on a clean install as this allows you to set the disk partition size to a particular value plus 10 MB. Increasing the size of the VMkcore partition does not resolve this issue.
Configuring a vSphere ESXi host to use a local USB device for VMkernel coredumps (broadcom.com) 
Configuring advanced options for ESXi/ESX (broadcom.com)
負荷の高い ESXi ホストで部分的なコア ダンプが生成される場合がある
工作负载较大的 ESXi 主机可能生成部分核心转储

Powered by Wolken Software