vmware-vpxd service crashes intermittently, generating core.vpxd-worker.##### under /var/core

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• The vpxd service crashes intermittently, generating a core.vpxd-worker.##### file in the /var/core directory each time.

• Rebooting the vCenter or manually restarting the vpxd service temporarily resolves the issue until the next crash.

• Further analysis indicates that the crash is related to host profile issues.

• Reviewing the /var/log/vmware/vpxd/vpxd.log file, repeated entries are observed before each core dump, leading to the vpxd crash.
  
  vpxd.log

[YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vpxLro opID=######] [VpxLRO] -- BEGIN task-57382002 -- hostprofile-####-- vim.profile.host.HostProfile.update -- <id>
[YYYY-MM-DDTHH:MM:SS] error vpxd[240211] [Originator@6876 sub=moHostProfile opID=#####] [GetValidationResult]: Validated Cache is empty.
[YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=#######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
-->    entity = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2e180321e8, h:380, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-######/vpxa>]: queryView
--> Received SOAP response fault from [<<io_obj p:0x000000ed0eec0768, h:21, <TCP '127.0.0.1 : 56547'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
--> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
--> }
[YYYY-MM-DDTHH:MM:SS] info vpxd[240211] [Originator@6876 sub=vmomi.soapStub[784] opID=######] SOAP request returned HTTP failure; <<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>, method: queryView; code: 500(Internal Server Error); fault: (vim.fault.InvalidName) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "Config.HostAgent.plugins.vimsvc.disableHostProfileApi",
-->    entity = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f2d0c1d7558, h:366, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#######/vpxa>]: queryView
--> Received SOAP response fault from [<<io_obj p:0x000000ed0f175078, h:18, <TCP '127.0.0.1 : 37157'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: queryView
--> 'Config.HostAgent.plugins.vimsvc.disableHostProfileApi' is invalid or exceeds the maximum number of characters permitted."
--> }

Resolution

Note: Ensure that you have a valid snapshot or backup of the vCenter before making any changes. If the affected vCenter is part of an ELM (Enhanced Linked Mode) setup, take powered-off snapshots for all vCenters in the ELM. If the affected vCenter is standalone, take a non-memory snapshot.

Follow the steps below:

1. Open a SSH session to the vCenter and connect to VCDB with the command:
   
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres


2. List the host profiles added to the vCenter:
   
   select profile_name,profile_id from vpx_profile;

Example Output: 
                      profile_name                       | profile_id
---------------------------------------------------------+------------
Default_DRS_HA_FT                                        |          9
Default_DRS_HA_FT                                        |         10
<profile_name>                                           |       1001
<profile_name>                                           |        601
<profile_name>                                           |        701
   
   
3. From the above list, note the host profile ID and its corresponding host profile name, which are noted in the vpxd.log.
   
   
4. Delete the problematic host profile:
   
delete from vpx_profile where profile_id='enter_id_noted_from_the_vpxd_log';


5. Restart vCenter services: 
   
   service-control --stop --all
service-control --start --all

Note: After implementing the above steps, the vpxd may crash again, referring to a different host profile the next time. You may have to repeat the above process till all the invalid host profiles are deleted from vCenter.