"no healthy upstream" error when accessing the vCenter Server UI

Products

• VMware vCenter Server

Issue/Introduction

• Accessing the vSphere UI via Browser fails with error "no healthy upstream".
• vCenter services status fails with vmware-vpxd service crash in specific:
  
  #service-control --status --all

Running:
applmgmt lookupsvc lwsmd observability-vapi pschealth vc-wsla-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-lic
ense vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd
mware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
StartPending:
vmware-sps vmware-updatemgr vmware-vsan-health wcp
Stopped:
observability vmcam vmonapi vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-vcha vmware-vdtc vmware-vpxd vstats
  
  
• Under /var/log/vmware/vmon/vmon.log, below error log snips are found:
  
  YYYY-MM-DDTHH:MM:SS In(05) host-###### Client info Uid=0,Gid=0,Pid=#######,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
YYYY-MM-DDTHH:MM:SS In(05) host-###### <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0
YYYY-MM-DDTHH:MM:SS In(05) host-###### Client info Uid=0,Gid=0,Pid=#######,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
YYYY-MM-DDTHH:MM:SS In(05) host-###### <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0
• Under /var/log/vmware/vpxd/vpxd.log, below error log snips are found:
  
  YYYY-MM-DDTHH:MM:SS info vpxd[#######] [Originator@#### sub=vmomi.soapStub[106]] SOAP request returned HTTP failure; <<cs p:############, TCP:localhost:####>, /sms/sdk>, method: selectCapabilities; code: 503(Service Unavailable); fault: (null)
YYYY-MM-DDTHH:MM:SS error vpxd[######] [Originator@#### sub=sms] [ConnectLocked] Failed to login to service: N7Vmacore24InvalidArgumentExceptionE(No version for VMODL calls to <<last binding: <<TCP '127.0.0.1 : #####'>, <TCP '127.0.0.1 : 1080'>> >,/sms/sdk>)
--> [context]################=[/context]
YYYY-MM-DDTHH:MM:SS info vpxd[#######] [Originator@#### sub=vpxdvpxdSignal] Received a shutdown request, exiting
YYYY-MM-DDTHH:MM:SS info vpxd[#######] [Originator@#### sub=Default] Initiating VMware VirtualCenter shutdown
YYYY-MM-DDTHH:MM:SS info vpxd[#######] [Originator@#### sub=Default] Shutting down VMware VirtualCenter
  
  
• Starting the vmware-vpxd service alone, fails and crashes immediately:
  

  # service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish ...
Performing start operation on service vpxd ...
Error executing start on service vpxd. Details {
"detail": [

  "id": "install.ciscommon. service.failstart",
"translatable": "An error occurred while starting service '$(0) s' ",
"args": [
"vpxd"
],

  "localized": "An error occurred while starting service 'vpxd'"

  ],

  "componentKey": null,
"problemId": null,
"resolution": null

  Service-control failed. Error: {
"detail": [

  "id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '$(0) s' ",
"args": [
"vpxd"

  "localized": "An error occurred while starting service 'vpxd'"

  "componentKey": null,
"problemId": null,
"resolution": null
}

• And a Core dump is created under /var/core directory.
• The command /usr/sbin/vpxd is run to check if the vpxd.cfg file is corrupt and it fails with the below error, along with Duplicate child ID that's causing the issue:
  
  # /usr/sbin/vpxd
Failed reading /etc/vmware-vpx/vpxd.cfg; N7Vmacore19FileFormatExceptionE (Duplicate child: costPerVmotionESX6x)
[context]######/###### == [/context]
[VpxdMain] Failed to init vmacore
Aborted (core dumped)

Cause

• The vpxd service crashes because the vmacore library encounters a FileFormatException while parsing the vpxd.cfg file.
• This is caused by duplicate <costPerVmotionESX6x> tags within the  <ResourceManager> tag leading to vmware-vpxd service termination due to conflicting entries.

Resolution

To resolve this issue, follow steps below:

Note: Take a snapshot of the vCenter Server Virtual Machine before implementing the steps below (Offline Snapshots of all the vCenter if they are in linked mode).

1. SSH to the vCenter Server as root user.

2. Create a backup for vpxd.cfg file, using below command:
   
   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

3. Open the vpxd.cfg file using a text editor, using below command:
   
   vi /etc/vmware-vpx/vpxd.cfg

4. Locate the <costPerVmotionESX6x> tag and delete it:
   
       <ResourceManager>
      <costPerVmotionESX6x>1</costPerVmotionESX6x>
      <costPerVmotionESX6x/>                                                           <-------------- remove this line
    </ResourceManager>

    

5. Save the changes, using below command:
   
   Press ESC and Type :wq! and Hit Enter.

6. Restart all the services using below command:
   
   service-control --stop --all && service-control --start --all