vCenter server service (vpxd) fails to start after reboot

Products

• VMware vCenter Server

Issue/Introduction

After rebooting the vCenter Server, the vpxd service fails to start. The issue presents with the following symptoms:

• The VMware vSphere Client does not load the UI and displays the error:
  
  Could not connect to one or more vCenter Server systems https://vCenterFQDN:443/sdk
  
  
• vpxd service does not start after reboot if checked from the SSH of the vCenter as a root user using: service-control --status --all

• No new entries appear in the /var/log/vmware/vpxd/ vpxd.log file.

• The /var/log/vmware/vmon/vmon.log file shows errors like:
  
  <vpxd> Service pre-start command's stderr: Failed to update memStopInMB in vpxd.cfg :xmlReadFd() failed  
<vpxd> Service pre-start command failed with exit code 1...

Cause

• The failure is due to an XML syntax error in the /etc/vmware-vpx/vpxd.cfg file.
• This issue can occur on vCenter versions particularly when manually configuring retreat mode Putting a Cluster in Retreat Mode using advanced settings.
• A faulty configuration may look like this:

<vcls>
  <clusters>
      <EntryID>
        <enabled>false</enabled>
      </EntryID>
        <domain-c123:Alpha-numericString>
  </clusters>
</vcls>

Note:- Only the portion before the colon (e.g., domain-c123) should be present. Including the full string (e.g., domain-c123:xyz) causes XML parsing errors.

Resolution

Ensure to take a snapshot of the vCenter before performing the below workaround Snapshot Best practices for vCenter Server Virtual Machines

To resolve the issue, you must correct the XML syntax in the vpxd.cfg file. Follow these steps carefully:

• Connect to the vCenter Server using SSH or the console via root credential.

• The vpxd.cfg file is located at:
  
  /etc/vmware-vpx/vpxd.cfg

• Take a backup of the file before making any changes:
  
  cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
  
  
• Open the file in a text editor:
  
  vi /etc/vmware-vpx/vpxd.cfg


• Look for the invalid <vcls> section, which may appear similar to:

<vcls>
  <clusters>
      <EntryID>
        <enabled>false</enabled>
      </EntryID>
        <domain-c123:Alpha-numericString>
  </clusters>
</vcls>

• Delete the entire <vcls> to </vcls> block, as it is causing the XML parsing error.

• Save and close the file using the wq! and press enter

• Restart the vpxd service to apply the changes:
  
  service-control --start vpxd

• Once the file is corrected, the vpxd service should start successfully.