"no healthy upstream" error when accessing the vCenter Server UI due to vpxd service crashing

Products

• VMware vCenter Server

Issue/Introduction

• Accessing the vSphere Client fails with error "no healthy upstream" on the Browser.
  
  
• Starting the vpxd service manually fails with the below error:
  
  root@vcenter [ ~ ]# service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

• The /var/run/log/vpxd/vpxd.log is not updated with the new log entries.
  
  
• The vpxd service crash creates core dumps in /var/core directory.
  
  
• The command /usr/sbin/vpxd is run to check if the vpxd.cfg file is corrupt and it fails with the below error:
  
  root@vcenter [ /var/core ]# /usr/sbin/vpxd
Failed reading /etc/vmware-vpx/vpxd.cfg; N7Vmacore19FileFormatExceptionE(Duplicate child: enabled)
[context]#####[/context]
|[VpxdMain] Failed to init vmacore
Aborted (core dumped)

Environment

VMware vCenter Server 8.x

Cause

• The vpxd service crashes because the vmacore library encounters a FileFormatException while parsing the vpxd.cfg file.
• This is caused by duplicate <enabled> tags within the <vcls> tag leading to vpxd service termination due to conflicting entries.

Resolution

NOTE: Take a snapshot of the vCenter Server Virtual Machine before implementing the steps below. In case of vCenter in Enhanced Linked Mode, follow VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

1. SSH to the vCenter Server as root. Enter shell.
   
   
2. Create a backup for vpxd.cfg file:
   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak


3. Open the file using a text editor:
   vi /etc/vmware-vpx/vpxd.cfg


4. Locate the <vcls> tag and delete the duplicate <enabled> tag:
     <vcls>
    <clusters>
      <domain-c#>
        <enabled>False</enabled>
        <enabled>False</enabled>                    <-------------- remove this line
         </domain-c#>
    </clusters>
  </vcls>
   
   
5. Save the changes:
   Hit ESC -> :wq!


6. Restart all the services on the vCenter Server:
   service-control --stop --all && service-control --start --all