vpxd service intermittently crashes with core dump (core.vpxd-worker), causing vCenter server UI will display blank inventory on vSphere 8.0 U1

Products

• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

• Unable to access the vCenter Server, the vpxd service will be in a stopped state.
• The vpxd service on the vCenter server crashes, generating a core dump with the file name "core.vpxd-worker"
• A manual restart of all vCenter server services would bring back the VC online.
  
  
• In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log you will see similar entries as below

vpxd-svcs.log.1:com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:8085 invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
vpxd-svcs.log.1:Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
vpxd-svcs.log.1:yyyy-mm-ddThh:mm:ss.mssZ [dataservice-7 [] WARN  com.vmware.cis.server.util.VpxdClient  opId=sps-DTCPoller-356856-896] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:8085 invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
vpxd-svcs.log.1:com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:8085 invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"

• In the /var/log/vmware/vmon/vmon.log you will see similar entries as below

yyyy-mm-ddThh:mm:ss.mssZ In(05) host-36347 <vpxd> Service is dumping core. Coredump count 2. CurrReq: 0
yyyy-mm-ddThh:mm:ss.mssZ Wa(03) host-36347 [ReadSvcSubStartupData] No startup information from vpxd.
yyyy-mm-ddThh:mm:ss.mssZ In(05) host-36347 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vpxd,UNHEALTHY,HEALTHY,1
yyyy-mm-ddThh:mm:ss.mssZ Wa(03) host-36347 <vpxd> Service exited. Exit code 1
yyyy-mm-ddThh:mm:ss.mssZ Wa(03) host-36347 <vpxd> Service exited unexpectedly. Crash count 2. Taking configured recovery action.
yyyy-mm-ddThh:mm:ss.mssZ Wa(03) host-36347 Failed to publish health status change.

Environment

vCenter Server 8.0 U1

Cause

There is a vulnerability in VPXD and all other VMACORE-based HTTP2 servers that allows any unauthenticated attacker to crash VPXD with a simple curl command. For the attack to be successful, someone needs to send a request to the server with a less common HTTP method, such as LOCK/UNLOCK.

Resolution

Issue is resolved in 8.0u1c.

To resolve this issue, patch to vCenter Server to the latest build in 8.0u3.