vCenter service crashes during host profile update tasks

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server goes down becomes unresponsive or disconnects.
• The vCenter UI reports "no healthy upstream" when crash happens. 
• The `vmware-vpxd` service crashes intermittently, specifically during Host Profile update task. 
• Core dump files named `core.vpxd-worker.####` are generated in `/var/core/`.
• In `/var/log/vmware/vpxd/vpxd.log` will indicate starting of the update host profile task. 

YYYY:MM:DDTHH:MM:SS.Z info vpxd[06914] [Originator@6876 sub=vpxLro opID=2ec183b1] [VpxLRO] -- BEGIN task-5272942 -- hostprofile-### -- vim.profile.host.HostProfile.update -- 521d9826-e484-1a7a-770b-8f0291d999dd(52030555-1eb6-241c-e6b7-ccd38abaf130) 

• The vmon log will indicate that the service has been crashed a after few minutes. 

YYYY:MM:DDTHH:MM:SS.Z Wa(03) host-2735 <vpxd> Service exited. Exit code 1
YYYY:MM:DDTHH:MM:SS.Z Wa(03) host-2735 <vpxd> Service exited unexpectedly. Crash count 0. Taking configured recovery action.
YYYY:MM:DDTHH:MM:SS.Z In(05) host-2735 <vpxd> Restarting service.

• When the above host profile is identified and downloaded, it will be revealed to be empty with 0 bytes. 
• This issue can impact multiple vCenter instances simultaneously if they share similar profile configurations or corruption.

Environment

VMware vCenter Server 8.0u3 

Cause

This issue occurs due to a null pointer dereference within the `vpxd` service. When vCenter attempts to validate a Host Profile that is missing a valid Reference Host or contains corrupted compliance metadata, the service fails to handle the exception gracefully and aborts.

Resolution

IMPORTANT: Take a file-based backup or an offline snapshot of the vCenter Server Appliance before proceeding.

To resolve the crash, identify and remove the corrupted profile:

1. Identify the Corrupted Profile: Review the `vpxd.log` immediately preceding the crash to identify the `hostprofile-####` triggered (e.g., `hostprofile-403`).
   1. Navigate to https://<vcenter_fqdn>/mob
   2. Navigate to Content > HostProfileManager
   3. Under Properties, locate the host profile ID hostprofile-403.
   4. The name of the profile will be listed adjacent to the profile id. 
2. Detach the Profile:
   1. Log in to the vSphere Client.
   2. Navigate to Menu > Policies and Profiles > Host Profiles.
   3. Right-click the suspected Host Profile and select Detach from Host/Cluster.
   4. Select all entities and click Detach.
3. Delete the Profile:
   1. Right-click the detached Host Profile and select Delete.
   2. Confirm the deletion to clear the corrupted entry from the vCenter Database.
4. Recreate the Profile:
   1. Select a known-compliant, healthy reference host.
   2. Right-click the host and select Host Profiles > Extract Host Profile.
   3. Attach this new profile to the environment.