vpxd service fails to start on vCenter due to a corrupt or invalid vpxd.cfg file

Products

VMware vCenter Server

Issue/Introduction

On vCenter, the vpxd service fails to start and creates no new log file upon each start attempt.

Environment

VMware vCenter Server

Cause

This happens if the /etc/vmware-vpx/vpxd.cfg file is corrupt or invalid.

The vpxd.cfg file can become corrupt due for a number of reasons:

vCenter virtual machine (VM) virtual disk problems.
Allowing virtual disks to fill up (other than /storage/archive) may result in a file or data corruption.
Doing a forced power off of vCenter VM, such as:
- From ESXi host crashing or being powered off abruptly.
- From selecting Power > Power Off or hitting the Power Off icon in the client.
- Running commands to kill the running VM processes via the ESXi command line interface.
Invalid changes to the file /etc/vmware/vpxd.cfg such as duplicate entries, typos, or malformed xml

Resolution

Check whether vpxd.cfg is corrupted

Connect to the VCSA command line using the root user via SSH connection or vCenter VM console
Run the following command to verify whether the vpxd.cfg file is valid.
- /usr/sbin/vpxd
  - Corrupt vpxd.cfg result: Aborted (core dumped).
  - Good vpxd.cfg result: a blank return.

Repair a corrupt vpxd.cfg

Deploy a new vCenter virtual machine of the same version.
- That is, if the currently affected vCenter is version 6.7 Update 1, deploy a new vCenter VM that is 6.7 Update 1.
- Note this VM is needed only temporarily for its copy of the vpxd.cfg file.
Get a copy of the vpxd.cfg file from the VM.

Note: access and copy files from the vCenter VM to a workstation using WinSCP or other SCP clients.

Copy this file to the affected vCenter to this location: /etc/vmware-vpx/vpxd.cfg.
Run the command to find out the current vCenter FQDN.

hostname -f

Run the command to find out the current vCenter Machine ID

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Run the command to find out the current vpostgres version

/opt/vmware/vpostgres/current/bin/psql -V

Run the command to find out the vCenter database password for vpxd

cat /etc/vmware-vpx/vcdb.properties

Copy down all of these findings to a notepad.
Use the vi editor to make changes to the new version of the vpxd.cfg file per Editing files on an ESXi host using vi except we will be editing the file on VCSA rather than ESXi.
- Remove the key_EvaluationExpiryDate
- Change hostnameUrl to be the correct vCenter FQDN found from the above instructions.
- Ensure all uri tags in the <sso> area point to the correct Platform Services Controller FQDN.
  - If this is a vCenter with an embedded PSC then this information will still be the vCenter's FQDN.

Example:

Change the machine ID for the vpxd solution user to the machine ID found from the above instructions

Example:

Ensure Postgres DB version found from the above instructions is correct.
If there is no entry for this in the vpxd.cfg file, this can be ignored.

Example:

Update the vcdb password with the password recorded earlier:
/sbin/vpxd -p

Run the command to start vpxd.

service-control --start vmware-vpxd

Additional Information

To gracefully power off a VCSA VM:

From the vCenter or ESXi host GUI

Right-click the VCSA VM and select Edit Settings...
Select Power > Shut down Guest OS

From the vCenter command line, accessed through the vCenter VM console or SSH session:

shutdown -h now

From the ESXi command line interface:

Find the vCenter's VM ID. The VM IDs are the first number for each line in the output
- vim-cmd vmsvc/getallvms
Shut down the VM
- vim-cmd vmsvc/power.shutdown VMID

Impact/Risks:
Corrupt vpxd.cfg will not allow vCenter Server to run which will not allow management changes to the vCenter site.
Following this procedure carries risks of causing further damage if not followed correctly.
Ensure a good backup of the vCenter before implementing these fixes.