Consumption of VMware Extended Guest Statistics on specific ESXi 5.5 releases or ESXi 6.0 may lead to virtual machine hanging issues
search cancel

Consumption of VMware Extended Guest Statistics on specific ESXi 5.5 releases or ESXi 6.0 may lead to virtual machine hanging issues

book

Article ID: 328498

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
When running applications, for example the SAP Host Agent 7.21 PL5, which make use of the latest ESXi Extended Guest Statistics within a virtual machine on an ESXi 5.5 Update 3 or ESXi 6.0 host, you may experience these symptoms in either Windows or Linux guest operating systems:
  • The virtual machine becomes hung irregularly.
  • In the esxtop performance monitoring utility, you observe that the memory usage statistics of the virtual machine world (OVHD) are steadily increasing.
  • In the the virtual machine log file vmware.log, you observe:

    YYYY-MM-DDT<time>Z| vcpu-1| I120: VERIFY bora/lib/misc/strutil.c:1079
    YYYY-MM-DDT<time>Z| vcpu-1| W110: A core file is available in "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001"
    YYYY-MM-DDT<time>Z| vcpu-1| W110: Writing monitor corefile "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmmcores.gz"
    YYYY-MM-DDT<time>Z| vcpu-1| W110: CoreDump error line 2160, error Cannot allocate memory</time></time></time></time>

    For more information about the location of virtual machine log files, see Locating virtual machine log files on an ESXi/ESX host (1007805)

  • In the /var/log/vmkernel.log file, you see entries similar to:　

    YYYY-MM-DDT<time>Z cpu21:101694)UserDump: 1907: Dumping cartel 100618 (from world 101694) to file /vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001 ...
    YYYY-MM-DDT<time>Z cpu20:101694)UserDump: 2027: Userworld coredump complete.
    YYYY-MM-DDT<time>Z cpu48:33427)WARNING: LinuxThread: 340: Error cloning thread: -28 (bad0081)
    YYYY-MM-DDT<time>Z cpu28:100618)WARNING: World: vm 100618: 3974: VMMWorld group leader = 100619, members = 2</time></time></time></time>


                Cause

                Recent enhancements in the Extended Guest Statistics collection allows the vendors to fetch metrics from within a guest operating system of a virtual machine. However, an issue has been found in the statistics collection in ESXi which leads to a memory leak and, in consequence, the virtual machine fails due to memory exhaustion.

                These collected statistics are, for example, heavily used in the context of SAP’s Host Agent (SAP Host Agent 7.21 PL5 and higher) and VMware vSphere (ESXi 5.5 Update 3 or ESXi 6.0).

                Resolution

                VMware understands the critical impact of this issue on its customers. The root cause has been identified.

                This is a known issue affecting VMware vSphere (ESXi 5.5 Update 3, ESXi minor releases after Update 3b, ESXi 6.0, ESXi 6.0u1, or ESXi 6.0u1a).

                This issue is resolved in:
                If you are unable to upgrade and you are running one of the affected versions of VMware vSphere ESXi listed above and the SAP Host Agent (SAP Host Agent 7.21 PL5 or later), VMware strongly recommends you open a support request to engage direct assistance by a VMware Technical Support Engineer to review the issue and implement the most appropriate resolution option. Mention this document to the responding Technical Support Engineer. For details on opening a support request, see How to file a Support Request in Customer Connect (2006985).

                To avoid experiencing this issue when using a third-party monitoring and statistics solution, postpone upgrading to vSphere 5.5 Update 3 or vSphere 6.0.x until an upgrade for your version of ESXi is available which fixes the issue. If you are already running ESXi 5.5 U3 or vSphere 6.0.x, your environment may not be affected if your statistics solution does not use VMware Extended Guest Statistics. Review the Administration guide and release notes of your third party statistics solution to see if the release you plan to install utilizes VMware Extended Guest Statistics. If yes, postpone the installation or upgrade of the solution or implement the workaround option until an upgrade for your version of ESXi is available which fixes the issue.

                To work around this issue, you may perform a vMotion migration of the virtual machine between ESXi hosts to temporarily clear the memory leak. VMware recommends that this migration should happen at least once each week. If your environment doesn't already migrate the affected virtual machines often enough, you can automate the migrations using a Schedule Task operation.

                A scheduled task can be used to periodically and automatically relocate the affected virtual machines. As stated, VMware recommends scheduling these tasks on a weekly basis. However, depending on the activity and load of the virtual machine(s), the scheduled task may need to be more frequent. For more information on creating a scheduled task for vMotion migration, see the Schedule Tasks section of the vSphere 6.0 vCenter Server and Host Management Guide.

                This article will be updated as soon as the regular patch release is available. You may subscribe to this article for RSS updates using the Subscribe to Document link in the Actions box.



                Additional Information

                For information in regards to the affected combination with SAP Host Agent 7.21 PL5, see the SAP Notes Related to VMware sub-article Note 2237937. This link requires a SAP account.

                Note: The preceding links were correct as of November 04, 2016. If you find a link is broken, provide feedback and a VMware employee will update the link.

                For more information on using esxtop to monitor performance issues, see Troubleshooting ESX/ESXi virtual machine performance issues (2001003).

                To be alerted when this document is updated, click the Subscribe to Article link in the Actions box..

                Locating virtual machine log files on an ESXi/ESX host
                How to download patches in Customer Connect
                Troubleshooting ESX/ESXi virtual machine performance issues
                How to file a Support Request in Customer Connect
                VMware ESXi 5.5, Patch Release ESXi550-201512001
                VMware ESXi 5.5, Patch Release ESXi550-201601501
                在特定 ESXi 5.5 版本或 ESXi 6.0 上使用 VMware 扩展客户机统计信息可能导致虚拟机挂起问题
                特定の ESXi 5.5 リリースまたは ESXi 6.0 での VMware Extended Guest Statistics の消費により仮想マシンのハングの問題が発生することがある
                VMware ESXi 5.5, Patch Release ESXi550-201602001
                VMware ESXi 5.5, Patch Release ESXi550-201608001
                VMware ESXi 5.5, Patch Release ESXi550-201609001
                VMware ESXi 5.5, Patch Release ESXi550-201612001

                Powered by Wolken Software