VPXD crashes due to a race condition in session logic

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The vpxd service crashes and creates a core.vpxd-worker file with a backtrace similar to the following. (Note the line corresponding to bora/vpx/vpxd/tagging/associations.cpp)

(gdb) bt
#0 0x00007f21bdd817ea in raise () from /tmp/ashrafmohame-debug-vc-core/cloudvm_libs_19338430/lib64/libc-2.28.so
#1 0x00007f21bdd82881 in abort () from /tmp/ashrafmohame-debug-vc-core/cloudvm_libs_19338430/lib64/libc-2.28.so
#2 0x00007f21c18af340 in Vmacore::System::SignalTerminateHandler (info=0x7f1eb6eb9ff0, ctx=0x7f1eb6eb9ec0) at bora/vim/lib/vmacore/posix/defSigHandlers.cpp:62
#3 <signal handler called>
#4 operator!=<const Vapi::Data::DataValue> (__a=...) at external/cayman_esx_toolchain_gcc9/usr/bin/../lib/gcc/x86_64-vmk-linux-gnu/9.3.0/../../../../x86_64-vmk-linux-gnu/include/c++/9.3.0/bits/shared_ptr.h:403
#5 Vpxd::Tagging::IsCalledFromTaggingService (activation=...) at bora/vpx/vpxd/tagging/associations.cpp:748
#6 0x000055d164b6d9fc in Vpxd::Vcenter::Tags::Vm::AssociationsSvc::AsyncServiceImpl::Get (this=<optimized out>, tag=..., activation=..., resultCb=...) at bora/vpx/vpxd/vcenter/tags/vm/associationsSvc/AsyncServiceImpl.cpp:84
#7 0x00007f21cb0cd812 in operator() (resultCb=..., activation=..., input=..., __closure=0x7f1f9cc5e8a0)
at external/cayman_esx_toolchain_gcc9/usr/bin/../lib/gcc/x86_64-vmk-linux-gnu/9.3.0/../../../../x86_64-vmk-linux-gnu/include/c++/9.3.0/bits/std_function.h:369
#8 operator()<Com::Vmware::Vcenter::Tags::Vm::AssociationsSvc::AsyncServiceBinderHelper::GetMethodBuilder::operator()(Com::Vmware::Vcenter::Tags::Vm::AssociationsSvc::AsyncServiceBinderHelper::GetMethodBuilder::FunctionType, const AsyncServiceInterfacePtr&, const LocalizerPtr&)::<lambda(const Com::Vmware::Vcenter::Tags::Vm::AssociationsSvc::GetMethod::Input&, const AsyncActivationPtr&, const ResultCbType&)>, Com::Vmware::Vcenter::Tags::Vm::AssociationsSvc::GetMethod::Input, std::shared_ptr<Vapi::Core::AsyncActivation>, Vapi::Provider::AsyncApiMethodResultFilter<std::set<std::basic_string<char> > > > (this=<optimized out>, resultCb=..., activation=..., input=..., slot=...)
at vapicpp/vapi-core/include/vapi/provider/AsyncApiMethodFilter.h:35
#9 Vapi::Provider::AsyncApiMethodFilter<Com::Vmware::Vcenter::Tags::Vm::AssociationsSvc::AsyncServiceBinderHelper::GetMethodBuilder::operator()
...

Environment

VMware vCenter Server 8.0.0
VMware vCenter Server 7.0.x

Cause

This is caused by a race condition in the session logic of vpxd.

Resolution

The issue is resolved in vCenter Server 7.0 Update 3l (build number 21477706)
The issue is resolved in vCenter Server 8.0b (build number 21216066)

Workaround:

Currently, the default session timeout is 30 minutes. By changing it to 31, the race condition causing vpxd to crash will be less likely to occur.

To accomplish this, add the following advanced setting to vCenter :

Configure->Settings->Advanced Settings you can add the following setting:

Name: config.vmacore.soap.sessionTimeout
Value: 31

Alternatively, you can add the following tag to /etc/vmware-vpx/vpxd.cfg and restart the vpxd service.