vmware-updatemgr and vmware-vsan-health services fails to start on vCenter Server 8.0

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• vSAN-Health Tab reports error:"vSAN-Health service is not running"
  
  
• /var/log/vmware/vsan-health/vmware-vsan-health-service.log reports below errors

2024-10-04T14:10:55.804Z WARNING vsan-mgmt[08188] [VsanVcExtension::__init__ opID=noOpId] Failed to log into VC, retrying in 10 seconds
2024-10-04T14:11:05.823Z WARNING vsan-mgmt[08188] [VsanVcExtension::__init__ opID=noOpId] Failed to log into VC, retrying in 10 seconds
2024-10-04T14:10:55.797Z warning vsanvcmgmtd[08840] [vSAN@6876 sub=HttpConnectionPool-000000 opId=08907f7e] Failed to get pooled connection; <cs p:00007f2d1c6d3760, TCP:localhost:80>, (null), duration: 3msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Bad Request)
-->  [context]zKq7AVECAQAAADXfTAEPdnNhbnZjbWdtdGQAANHHVGxpYnZtYWNvcmUuc28AAGEKRgBRBkcAdc1MACqqSgB9GksAer5GADc9OwB0PjsACD87AImaOwAS6jsA4rZSAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-04T14:10:55.802Z warning vsanvcmgmtd[08188] [vSAN@6876 sub=Py2CppStub opId=08907f7e] |- EExit LOCAL::vim.SessionManager.loginExtensionByCertificate (8 ms)
2024-10-04T14:10:55.802Z warning vsanvcmgmtd[08188] [vSAN@6876 sub=Py2CppStub opId=08907f7e] Exception while invoking VMOMI method 'LOCAL::vim.SessionManager.loginExtensionByCertificate': N7Vmacore4Http13HttpExceptionE(HTTP error response: Bad Request)
--> [context]zKq7AVECAQAAADXfTAEPdnNhbnZjbWdtdGQAANHHVGxpYnZtYWNvcmUuc28AAGEKRgBRBkcAdc1MACqqSgB9GksAer5GADc9OwB0PjsACD87AImaOwAS6jsA4rZSAYd/AGxpYnB0aHJlYWQuc28uMAACLzYPbGliYy5zby42AA==[/context]
2024-10-04T14:11:05.816Z info vsanvcmgmtd[08188] [vSAN@6876 sub=PyCppVmomi] Loaded system certificate from VECS.

• /var/log/vmware/envoy/envoy.log Reports below error

2024-12-06T10:12:08.339Z error envoy[2451] [Originator@6876 sub=config] listener 'edge_http_v4' failed to bind or apply socket options: cannot bind '0.0.0.0:80': Address already in use
2024-12-06T10:12:08.347Z info envoy[2451] [Originator@6876 sub=upstream] lds: add/update listener 'direct_response_v4'

•  contents of /var/log/vmware/rhttpproxy/rproxy_init.log.stderr:

Starting service process with pid: 25818.
E0115 10:30:37.045973004   25855 socket_utils_common_posix.cc:223] check for SO_REUSEPORT: {"created":"@1736937037.045959142","description":"SO_REUSEPORT unavailable on compiling system","file":"/build/mts/release/bora-20889737/cayman_grpc/grpc/src/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":192}

 

Environment

vCenter Server 8.0.1

Cause

This issue occurs when Envoy service is unable to take ownership of port 80 because it is already being used by the httpd process

Resolution

To resolve this issue we need to identify the httpd service using port 80 and kill/stop the process.

We can execute below commands: 

• Take snapshot of VCSA VM without memory ( Powered off Snapshot if VC nodes are in ELM)

• To identify the service/process id for port 80
netstat -nlp | grep ':80 ' 

• Execute the below command to disable the httpd service from starting during the boot process. This will prevent port 80 from being used by the httpd process
systemctl disable --now httpd

• Then reboot the vCenter and check if envoy is now using the port 80 by executing command:
  netstat -nlp | grep ':80 '