vmware-rhttpproxy service fails to start with error "Application error: No endpoints found" in vCenter server appliance

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter UI is not accessible.
• Multiple vCenter services fails to start
• vmware-rhttpproxy service fails to start with the log entries similar to the below:

/var/log/vmware/rhttpproxy/rhttpproxy.log

2021-04-14T15:36:44.610Z info rhttpproxy[35477] [Originator@6876 sub=Default] Listening on HTTPS port 443.
2021-04-14T15:36:44.610Z error rhttpproxy[35477] [Originator@6876 sub=Default] Application error: No endpoints found; <resolver p:0x00005644f51112a0, 'localhost:549'>
2021-04-14T15:36:44.611Z error rhttpproxy[35477] [Originator@6876 sub=Default] Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.7.0, build: build-17137320, tag: rhttpproxy, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B0D04]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B2524]: Vmacore::System::SystemFactory::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00189CFE]: Vmacore::Throwable::Throwable(std::string&&)
--> backtrace[03] libvmacore.so[0x0025F54A]
--> backtrace[04] libvmacore.so[0x00265C98]
--> backtrace[05] rhttpproxy[0x00039AEA]
--> backtrace[06] rhttpproxy[0x0003770D]
--> backtrace[07] libc.so.6[0x000205C0]
--> backtrace[08] rhttpproxy[0x00037B2D]
--> [backtrace end]

 

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x

Cause

This issue may occur if vCenter Server is missing loopback entry in  "/etc/hosts"  file or invalid content in the /etc/hosts

To confirm the cause:

• Connect to the VCSA through ssh
• # less /etc/hosts

# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
10.##.##.##  vcenter01.domain.com vcenter01 localhost 
::1 vcenter01.domain.com vcenter01 localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

Resolution

To fix this issue, /etc/hosts file needs to be changed by removing incorrect loopback address and add the correct one.

Before applying the resolution steps, make sure you have vCenter proper backups or offline snapshot (Snapshot with vCenter VM powered off)

1. Connect to VCSA through SSH
2. # cd /etc/
3. Backup the hosts file using the below command

# cp hosts hosts.bk

4. Make the changes

• # vi hosts
• Type i to enter insert mode
• Delete the IP address beside the vCenter FQDN -->  10.##.##.## in this example
• add 127.0.0.1 instead to look like

# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1  vcenter01.domain.com vcenter01 localhost
::1 vcenter01.domain.com vcenter01 localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

• Press ESC to exit insert mode
• Press !wq to save and exit.

5. Start vCenter services using the below command:

#service-control --start --all