The vSphere UI is not accessible, displaying the error: “No healthy upstream.” and vmware-sps and vmware-vpxd failed to start.

Products

• VMware vCenter Server

Issue/Introduction

• vCenter UI is not accessible with error "No healthy upstream" and vmware-vpxd and sps failed to start. 
• While starting the vpxd and sps service it failed with below error.

YYYY-MM-DDThh:mm:ss Running command: ['/usr/lib/vmware-vmon/vmon-cli', ' -- batchstart', 'ALL' ]
YYYY-MM-DDThh:mm:ss Done running command
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, updatemgr, wcp, vpxd, vsan-health, vstats servic
es. Error: Operation timed out

• Logs /var/log/vmware/vpxd/vpxd.log shows errors similar to below snippets:

YYYY-MM-DDThh:mm:ss warning vpxd[66166] [Originator@6876 sub=Vmomi opID=vsan-PC-646c4c6ccafe5-a7] VMOMI activation LRO failed; << 5223f756-9409-24b9-f721-aa3ed3574a85, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 52696'>>, CustomFieldsManager, vim. CustomFieldsManager. addFieldDefinition, <vim. version. v8_0_3_0, internal, 8.0.3.0>, {stm: {<io_obj p:0x00007fec6877f790, h:171, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 52696'>>, id: 239, state(in/out): 3/1}, session: <5223f756-9409-24b9-f721-aa3ed3574a85, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 52696'>>, req: {POST, /sdk} }>, N3Vim5Fault13DuplicateName9ExceptionE(Fault cause: vim. fault.DuplicateName

Environment

vCenter Server 8.x

Cause

The issue occurs because the mo_type_name field in vpx_field_def table is null for the entry com.vmware.vcenter.network.folder.externalManaged.

VCDB=# select id,name,mo_type_name from vpx_field_def where name='com.vmware.vcenter.network.folder.externalManaged';
 id  |                       name                        | mo_type_name
-----+---------------------------------------------------+--------------
 194 | com.vmware.vcenter.network.folder.externalManaged |
(1 row)

Resolution

Before making any changes refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Following steps has to be performed in 7.x vCenter prior to upgrade:

1. Stop vCenter Server service

service-control --stop vpxd

2. Connect to vCeneter Server Database:

psql -d VCDB -U postgres

3. Update vpx_field_def table to correct null mo_type_name value:

update vpx_field_def set mo_type_name='vim.Folder' where name='com.vmware.vcenter.network.folder.externalManaged';

4. Start vCenter Service

service-control --start vpxd