HCX Manager 9443 UI becomes inaccessible following the deletion of the vCenter configuration

Products

• VMware HCX

Issue/Introduction

• After deleting/unregister the vCenter configuration from HCX Manager (9443 Admin UI > Configuration > vCenter), the Admin UI on port 9443 becomes inaccessible after a reboot.
• Authentication to HCX Manager 443 hybridity UI fails.
• HCX Manager 9443 Admin UI page does not load.

• Running the following command confirms the appliance-management service failed to start due to the web-engine service not running and the service is stuck in an "activating" state:

  root@hcx-mgr [ ~ ]# systemctl status appliance-management
● appliance-management.service - Appliance Management
Loaded: loaded (/etc/systemd/system/appliance-management.service; enabled; vendor preset: enabled)
Active: activating (start-pre) since ### YYYY-MM-DD HH:MM:SS UTC; #min ago
Cntrl PID: 2727 (service-depende)
Tasks: 2 (limit: 2385)
Memory: 424.0K
CGroup: /system.slice/appliance-management.service
├─2727 /bin/bash /etc/systemd/service-dependency-check.sh ignore-service-failures postgresdb zookeeper kafka app-engine web-engine
└─3213 sleep 30
MMM DD HH:MM:SS <hcx-mgr> service-dependency-check.sh[2727]: web-engine is not running.
MMM DD HH:MM:SS <hcx-mgr> service-dependency-check.sh[2727]: web-engine is not running.

• However, checking the status of web-engine confirms that it is running:
  
  root@hcx-mgr [ ~ ]# systemctl status appliance-management
● web-engine.service - WebEngine
Loaded: loaded (/etc/systemd/system/web-engine.service; enabled; vendor preset: enabled)
Active: active (running) since ### YYYY-MM-DD HH:MM:SS UTC; #min ago


• The HCX Manager log (/common/logs/appliance-management/appliance-management-startup.log) indicates that the vCenter configuration cannot be found.
  <timestamps> UTC [https-jsse-nio-9443-exec-10, , ] WARN  c.v.h.admin.config.ApplianceUtil- ApplianceConfig section: vcenter not found.
  
  
• From the HCX Manager log  (/common/logs/appliance-management/appliance-management.log), a vCenter configuration delete request was detected. This action caused the appliance management service to fail.
  
  <timestamps> UTC [https-jsse-nio-9443-exec-10, , ] DEBUG o.s.web.servlet.DispatcherServlet- DELETE "/api/admin/global/config/vcenter/####-####-####", parameters={}
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] DEBUG o.s.w.s.m.m.a.RequestMappingHandlerMapping- Mapped to com.vmware.hybridity.admin.config.ServiceConfigRestController#deleteConfig(String, String)
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] INFO  c.v.h.admin.config.ServiceConfigDAO- Deleted item: ####-####-####, section: vcenter
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] INFO  c.v.h.admin.config.ServiceConfigDAO- Delete request: ####-####-#### completed. Publish to kafka delete config change: {"data":[],"section":"vcenter","eventType":"MODIFY"}
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] INFO  c.v.h.a.c.ApplianceConfigValidator- Removing VC node with id ####-####-#### from endpointlink
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] INFO  c.v.v.h.dao.EndPointLinkUtils- NODE To be deleted is com.vmware.vchs.hybridity.adapters.postgresdb.querybuilder.DeleteStatement$WhereClause@508a9977
<timestamps> UTC [https-jsse-nio-9443-exec-10, , ] INFO  c.v.v.h.dao.EndPointLinkUtils- DELETE RESULT : {"result":{"n":1},"status":"SUCCESS"}

Environment

VMware HCX

Cause

This is an expected behavior. The appliance-management service, which listens on port 9443, depends on the web-engine service. Since the web-engine service is integrated with vCenter/SSO, it will not function properly without a valid configuration, leading to failed authentication on the HCX hybridity UI 443 page.

Resolution

If you believe you are experiencing this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.

Additional Information

NOTE: Do not remove vCenter from HCX Admin 9443 UI