vmware-vpxd service crashes intermittently, generating core.vpxd-worker.##### under /var/core

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• The vpxd service crashes intermittently, generating a core.vpxd-worker.##### file in the /var/core directory each time.

• Rebooting the vCenter or manually restarting the vpxd service temporarily resolves the issue until the next crash.

• Further analysis indicates that the crash is related to host profile issues.

• Reviewing the /var/log/vmware/vpxd/vpxd.log file, repeated entries are observed before each core dump, leading to the vpxd crash.
  
  vpxd.log
  
  

  [YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vpxLro opID=######] [VpxLRO] -- BEGIN task-57382002 -- hostprofile-####-- vim.profile.host.HostProfile.update -- <id>
  [YYYY-MM-DDTHH:MM:SS] error vpxd[240211] [Originator@6876 sub=moHostProfile opID=#####] [GetValidationResult]: Validated Cache is empty.
  [YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=#######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
  -->    entity = <unset>
  -->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-######/vpxa>]: queryView
  --> Received SOAP response fault from [<<io_obj p:0x000000ed0eec0768, h:21, <TCP '127.0.0.1 : 56547'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
  --> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
  --> }
  [YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
  -->    entity = <unset>
  -->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>]: queryView
  --> Received SOAP response fault from [<<io_obj p:0x000000ed0f175078, h:18, <TCP '127.0.0.1 : 37157'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
  --> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
  --> }

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

The crash is triggered by corrupt or inconsistent entries in the vCenter Server Database (VCDB) within the Host Profile tables. When vpxd attempts to validate or update these profiles, it receives an unexpected SOAP fault from the host agent (vpxa), leading to an unhandled exception and subsequent service crash.

Resolution

Note: Ensure that you have a valid snapshot or backup of the vCenter before making any changes. If the affected vCenter is part of an ELM (Enhanced Linked Mode) setup, take powered-off snapshots for all vCenters in the ELM. If the affected vCenter is standalone, take a non-memory snapshot.

Follow the steps below:

1. Open a SSH session to the vCenter and connect to VCDB with the command:
   

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
   

2. List the host profiles added to the vCenter:
   
   

   select profile_name,profile_id from vpx_profile;
   
   Example Output: 
                         profile_name                       | profile_id
   ---------------------------------------------------------+------------
   Default_DRS_HA_FT                                        |          9
   Default_DRS_HA_FT                                        |         10
   <profile_name>                                           |       1001
   <profile_name>                                           |        601
   <profile_name>                                           |        701

   
   
   
3. From the above list, note the host profile ID and its corresponding host profile name, which are noted in the vpxd.log.
   
   
4. Delete the problematic host profile:
   

   delete from vpx_profile where profile_id='enter_id_noted_from_the_vpxd_log';
   

5. Restart vCenter services: 
   
   

   service-control --stop --all
   service-control --start --all

Note: After implementing the above steps, the vpxd may crash again, referring to a different host profile the next time. You may have to repeat the above process till all the invalid host profiles are deleted from vCenter.