VMware vCenter Server Appliance vpxd service continually backtraces due to ESXi host port exhaustion
search cancel

VMware vCenter Server Appliance vpxd service continually backtraces due to ESXi host port exhaustion

book

Article ID: 320792

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • The vpxd service does not stop after the backtrace
  • On the vCenter Server Appliance the /storage/core/ is 100% full
  • Cannot enable SSH on ESXi host that has run out of ports
  • In the /var/log/vmware/vpx/vpxd.log file, you see backtraces similar to:

    YYYY-MM-DD TIME [7F6A014A9700 trivia 'HttpConnectionPool-000001' opID=HB-host-1845@88205-16301ff9] [IncConnectionCount] Number of connections to <cs p:00007f6a0d0f46c0, TCP:esxi6.gsslabs.org:443> incremented to 1
    YYYY-MM-DD TIME [7F6A014A9700 trivia 'HttpConnectionPool-000001' opID=HB-host-1845@88205-16301ff9] [PopPendingConnection] Found pending connection to<cs p:00007f6a0d0f46c0, TCP:<i>host_fqdn:443>
    YYYY-MM-DD TIME [7F6A014A9700 trivia 'vmomi.soapStub[967]' opID=HB-host-1845@88205-16301ff9] Request started [N7Vmacore4Http13UserAgentImpl22AsyncSendRequestHelperE:0x7f6a0d339170]
    YYYY-MM-DD TIME [7F6A080E0700 trivia 'ThreadPool'] ThreadPool[idle:245, busy_io:0, busy_long:17] HandleWork(type: 0, fun:N5boost3_bi6bind_tINS0_11unspecifiedENS_8functionIFvRKNS_6system10error_codeENS_4asio2ip23basic_resolver_iteratorINS9_3tcpEEEEEENS0_5list2INS0_5valueIS5_EENSG_ISC_EEEEEE)
    YYYY-MM-DD TIME [7F6A080E0700 trivia 'ThreadPool'] HandleWork() leaving
    YYYY-MM-DD TIME [7F6A02D5A700 trivia 'ThreadPool'] ThreadPool[idle:245, busy_io:0, busy_long:17] HandleWork(type: 0, fun: N5boost3_bi6bind_tINS0_11unspecifiedENS_8functionIFvRKNS_6system10error_codeEEEENS0_5list1INS0_5valueIS5_EEEEEE)
    YYYY-MM-DD TIME [7F69FF469700 verbose 'vpxservicesvpxdTaskInfo'] backtrace[00] rip 00007f6a1d66bfa4 Vmacore::System::Stacktrace::CaptureWork(unsignedint)
    --> backtrace[01] rip 00007f6a1d5560c2 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
    --> backtrace[02] rip 00007f6a1d4b7835 Vmacore::Throwable::Throwable(std::string const&)
    --> backtrace[03] rip 00007f6a249a9ea4 /usr/lib/vmware-vpx/vpxd(+0x1121ea4) [0x7f6a249a9ea4]
    --> backtrace[04] rip 00007f6a2499e7a4 /usr/lib/vmware-vpx/vpxd(+0x11167a4) [0x7f6a2499e7a4]
    --> backtrace[05] rip 00007f6a249930ba /usr/lib/vmware-vpx/vpxd(+0x110b0ba) [0x7f6a249930ba]
    --> backtrace[06] rip 00007f6a2499f8c3 /usr/lib/vmware-vpx/vpxd(+0x11178c3) [0x7f6a2499f8c3]
    --> backtrace[07] rip 00007f6a1dd1b93f Vmomi::StubImpl::_Invoke_Task(Vmomi::ManagedMethod*, Vmacore::RefVector<Vmomi::Any>&, Vmacore::Ref<Vmomi::Any>&)
    --> backtrace[08] rip 00007f6a1de16c94 Vmomi::Core::PropertyCollectorStub::WaitForUpdates(Vmacore::Optional<std::string> const&, Vmacore::Ref<Vmomi::Core::PropertyCollector::UpdateSet>&)
    --> backtrace[09] rip 00007f6a25bcc102 /usr/lib/vmware-vpx/vpxd(+0x2344102) [0x7f6a25bcc102]
    --> backtrace[10] rip 00007f6a25bccc54 /usr/lib/vmware-vpx/vpxd(+0x2344c54) [0x7f6a25bccc54]
    --> backtrace[11] rip 00007f6a1d5c22de /usr/lib/vmware-vpx/libvmacore.so(+0x3492de) [0x7f6a1d5c22de]
    --> backtrace[12] rip 00007f6a1d5c2d5a /usr/lib/vmware-vpx/libvmacore.so(+0x349d5a) [0x7f6a1d5c2d5a]
    --> backtrace[13] rip 00007f6a1d5c312a /usr/lib/vmware-vpx/libvmacore.so(+0x34a12a) [0x7f6a1d5c312a]
    --> backtrace[14] rip 00007f6a1d5ca9b7 /usr/lib/vmware-vpx/libvmacore.so(+0x3519b7) [0x7f6a1d5ca9b7]
    --> backtrace[15] rip 00007f6a1d5c7d4f /usr/lib/vmware-vpx/libvmacore.so(+0x34ed4f) [0x7f6a1d5c7d4f]
    --> backtrace[16] rip 00007f6a1d5bf46b /usr/lib/vmware-vpx/libvmacore.so(+0x34646b) [0x7f6a1d5bf46b]
    --> backtrace[17] rip 00007f6a1d659edf /usr/lib/vmware-vpx/libvmacore.so(+0x3e0edf) [0x7f6a1d659edf]
    --> backtrace[18] rip 00007f6a1bfd07f6 /lib64/libpthread.so.0(+0x77f6) [0x7f6a1bfd07f6]
     
  • In the /var/log/vpxa.log file, you see entries similar to:

    YYYY-MM-DD TIME [3DA94B70 verbose 'Default'] [CreateHttpSession] Out of HTTP sessions


Environment

VMware vSphere ESXi

Cause

This issue occurs when all ports are being exhausted on an ESXi host that is connected to the vCenter Server Appliance due to connections not being closed after use by third-party products or APIs. vCenter Server remains running though communication to the affected ESXi hosts are degraded.

Resolution

To resolve this issue, the management agents are required to be restarted on the affected ESXi host.

For more information, see Restarting the Management agents in ESXi.



Additional Information

Powered by Wolken Software