"No Healthy Upstream" when trying to access vSphere UI - Multiple services fails to start

Products

• VMware vCenter Server

Issue/Introduction

• "No Healthy Upstream" when trying to access vSphere UI - Multiple services fails to start due to incorrect configuration in the /etc/hosts file.
  
  
• vpxd and vpxd-svcs services on vCenter fails to start.
  
  
• The error message in vpxd and vpxd-svcs indicates that there are attempts to connect to the vCenter vPostgres database using the external IP address of the VCSA, instead of the local loopback address.
  
  
• Following error message can be found in /var/log/vmware/vpxd-svcs/vpxd-svcs.log:

YYYY-MM-DDTHH:MM:SS [main ERROR com.vmware.cis.core.kv.impl.Provider.VCDBProviderFactory opId=] SQL Error:org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: no pg_hba.conf entry for host "<Host_IP_Address>", user "vc", database "VCDB", SSL off)
  
  
• Following error message may also be present in /var/log/vmware/vpxd/vpxd.log:
  
  YYYY-MM-DDTHH:MM:SS error vpxd[06815] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Failed to connect to database: ODBC error: (08001) - [unixODBC]FATAL: no pg_hba.conf entry for host "<Host_IP_Address>", user "vc", database "VCDB", SSL off

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

• vpxd service or other dependent services fails to start due to incorrect or missing entries in the /etc/hosts file on the vCenter Server Appliance

• /etc/hosts file must be configured correct for vCenter services to function as expected.
  
  
• The file should be structured as follows:
  

  # Begin /etc/hosts (network card version)

  # End /etc/hosts (network card version)
# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  <vCenter's FQDN> <vCenter's shortname> localhost
::1  <vCenter's FQDN> <vCenter's shortname> localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

• Note: The IP address used must be the loopback address (127.0.0.1 or ::1), not the vCenter server's actual IP address.

Resolution

• Create an offline snapshot of the vCenter.
  
  Note: If the vCenter Server is part of a Linked Mode, VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice
  
  
• SSH into the vCenter 
  
  
• Make a backup copy of the /etc/hosts file.
  
  cp /etc/hosts /var/core/hosts.bak
  
  
• Manual modification of /etc/hosts file is not recommended. Use the following command to re-enter hostname and IP address details.

/opt/vmware/share/vami/vami_config_net


• • Select Option 3 and Option 6 to re-enter Hostname and IP address:

root@<VC-Name> [ ~ ]# /opt/vmware/share/vami/vami_config_net

 Main Menu

0)      Show Current Configuration 
1)      Exit this program
2)      Default Gateway
3)      Hostname
4)      DNS
5)      Proxy Server
6)      IP Address Allocation for eth0
Enter a menu number [0]:

• Verify that the /etc/hosts file includes the correct and valid entries. A sample structure is available in the Cause section of this article for reference.
  
  
• Restart all services by running the below command:
  
  service-control --stop --all && service-control --start --all