vCenter VPXD service crashes randomly due to IOFilters

Products

• VMware vCenter Server 8.0

Issue/Introduction

• The vCenter vpxd service crashes resulting in creation of core dump files.
• A restart of service temporarily fixes the issue and the service crashes again while performing any task which involves IOFilters

ls /var/core/

core.vpxd-worker.<vpxd_pid>

 

• The logs end abruptly with reconfigure task

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SSZ+00:00 - time the service was last started YYYY-MM-DDTHH:MM:SSZ, Section for VMware VirtualCenter, pid=<vpxd_pid>, version=8.0.3, build=24853646, option=Release

YYYY-MM-DDTHH:MM:SSZ+00:00 info vpxd[04560] [Originator@6876 sub=vpxLro opID=#######-####-##] [VpxLRO] -- BEGIN lro-114230 -- vm-#### -- vim.VirtualMachine.reconfigure -- <session_id>


/var/log/vmware/vmon.log

YYYY-MM-DDTHH:MM:SSZ+00:00 Wa(03) host-#### <vpxd> Service exited. Exit code 1
YYYY-MM-DDTHH:MM:SSZ+00:00 Wa(03) host-#### <vpxd> Service exited unexpectedly. Crash count 3. Taking configured recovery action.

Environment

vCenter 8.x

Cause

Validating the ESXi logs, confirms that the Class for iofilter is unknown

less /commands/esxcfg-info.txt

\==+HostIOFilter :
      \==+IO Filter :
         |----Name..................................................########
         |----Vendor................................................EMC
         |----Version...............................................6020.m.89-1OEM.700.1.0.15843807
         |----ID....................................................EMC_bootbank_#########_6020.m.89-1OEM.700.1.0.15843807
         |----LocalID...............................................emcsplitter
         |----Class.................................................unknown
         |----Release Date..........................................YYYY-MM-DDTHH:MM:SSZ+00:00
         |----Operational...........................................Yes
         |----Summary...............................................EMC RP  #########;[ESX_UUID,mirror_sbr,ip_#####,shared_vmdks]
      \==+IO Filter :
         |----Name..................................................#########
         |----Vendor................................................EMC
         |----Version...............................................6010.m.45-1OEM.700.1.0.15843807
         |----ID....................................................EMC_bootbank_##########_6010.m.45-1OEM.700.1.0.15843807
         |----LocalID...............................................emcjiraf
         |----Class.................................................unknown
         |----Release Date..........................................YYYY-MM-DDTHH:MM:SSZ+00:00
         |----Operational...........................................Yes
         |----Summary...............................................EMC RP #####;[#######]

Resolution

The iofilters are set to an "unknown" type, which is not supported as per the official documentation, Types of I/O Filters.

Contact the IOFilter vendor for further assistance

As a workaround, the IOFilters can be uninstalled which helps in keeping the vpxd stable.