vmware-vpxd service crashes intermittently, generating core.vpxd-worker.##### under /var/core

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• The vmware-vpxd service crashes intermittently, generating a core.vpxd-worker.##### file in the /var/core directory each time.

• Rebooting the vCenter or manually restarting the vmware-vpxd service temporarily resolves the issue until the next crash.

• Further analysis indicates that the crash is related to host profile issues.

• Reviewing the /var/log/vmware/vpxd/vpxd.log file, repeated entries are observed before each core dump, leading to the vpxd crash.

• vpxd.log
  
  [YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vpxLro opID=######] [VpxLRO] -- BEGIN task-57382002 -- hostprofile-####-- vim.profile.host.HostProfile.update -- <id>
[YYYY-MM-DDTHH:MM:SS] error vpxd[240211] [Originator@6876 sub=moHostProfile opID=#####] [GetValidationResult]: Validated Cache is empty.
[YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=#######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
-->    entity = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-######/vpxa>]: queryView
--> Received SOAP response fault from [<<io_obj p:0x000000ed0eec0768, h:21, <TCP '127.0.0.1 : 56547'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
--> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
--> }
[YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
-->    entity = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>]: queryView
--> Received SOAP response fault from [<<io_obj p:0x000000ed0f175078, h:18, <TCP '127.0.0.1 : 37157'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
--> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
--> }

Resolution

Note: Ensure that you have a valid snapshot or backup of the vCenter before making any changes. If the affected vCenter is part of an ELM (Enhanced Linked Mode) setup, take powered-off snapshots for all vCenters in the ELM. If the affected vCenter is standalone, take a non-memory snapshot.

To resolve the issue, follow the steps below:

• Connect to VCDB with the command:
  
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres


• List the host profiles added to the vCenter:
  
  select profile_name,profile_id from vpx_profile;

Example Output: 

                      profile_name                       | profile_id

---------------------------------------------------------+------------

Default_DRS_HA_FT                                       |          9

Default_DRS_HA_FT                                       |         10

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx     |       1001

yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy |        601

zzzzzzzzzzzzzzzzzzzzzzzzzzzzz                           |        701

 

• From the above list, note the host profile ID and its corresponding host profile name, which are noted in the vpxd.log.
• Delete the problematic host profile:
  
delete from vpx_profile where profile_id="enter id noted from the vpxd logs";


• Restart vCenter services: 
  
  service-control --stop --all
service-control --start --all



Note: After implementing the above steps, the vpxd may crash again, referring to a different host profile the next time. You may have to repeat the above process till all the invalid host profiles are deleted from vCenter.