vCenter Server UI displays "no healthy upstream" due to vpxd service failing to start

Products

• VMware vCenter Server

Issue/Introduction

• Accessing vCenter UI returns "no healthy upstream" error.

• Attempting to start all the services manually using the following command fails to start vpxd service:
  
  service-control --start --all

• vpxd services status:

root@vcenter [/]# service-control --status
Running:
 applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
StartPending:
 wcp
Stopped:
 vmcam vmonapi vmware-content-library vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-vcha vmware-vpxd vmware-vsan-health vstats

• The following log entries are observed in /var/log/vmware/vmon/vmon.log

YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 Received start request for vpxd
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/py/vpxd-prestart.py
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd> Service pre-start command completed successfully.
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd> Constructed command: /usr/sbin/vpxd
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd> Running the API Health command as user vpxd
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vpxd -f /var/run/vpxd/vmware-vpxd-health-status.xml
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 Client info Uid=0,Gid=0,Pid=221072,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
YYYY-DD-HHTHH:MM:SSZ In(05) host-2687 <vpxd> Service is dumping core. Coredump count 0. CurrReq: 1

• /sbin/vpxd command in a VCSA shell session returns "Aborted (core dumped)"

Failed reading /etc/vmware-vpx/vpxd.cfg; N7Vmacore19FileFormatExceptionE(Duplicate child: useSSL)
[VpxdMain] Failed to init vmacore
Aborted (core dumped)

Environment

• VMware vCenter Server 7.0.x
• VMware vCenter Server 8.0.x

Cause

• This problem arises from invalid or duplicate entries found in the vpxd configuration file located at /etc/vmware-vpx/vpxd.cfg, specifically related to SSL for NFC data traffic.

• Incorrect configuration when enabling or disabling vSphere SSL for the provisioning of NFC data traffic through the vCenter UI may result in a duplicate entry in vpxd.cfg.

Resolution

NOTE: Before applying the steps below, take a backup or a snapshot of the vCenter Server Appliance. If the vCenter is part of a Linked Mode replication setup, backup/snapshot all replicating nodes in powered off state. Refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

1. SSH in to vCenter Appliance

2. Verify if duplicate entry for "useSSL" or "maxConcurrentSyncItems" is present /etc/vmware-vpx/vpxd.cfg using vi editor

   <nfc>
    <useSSL>false</useSSL>

       <useSSL>false</useSSL>

   </nfc>

3. If a duplicate entry is present, delete the duplicate entry using VI editor.

4. Start the all the services on the vCenter Server with the command service-control --start --all

Additional Information

Disabling SSL for NFC data traffic in vCenter Server: Disabling SSL for NFC data traffic in vCenter Server