vmware-vpxd service crash with core dump "core.vpxd-worker.#######" on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• You will observe core dumps generated for each crash in the /var/core  core.vpxd-worker.#######.

• /var/log/vmware/vpxd/vpxd.log will give us below error:

  YYYY-MM-DDThh:mm:ss error vpxd[34811] [Originator@6876 sub=Memory checker] Current value ####### exceeds hard limit #######. Shutting down process.
  YYYY-MM-DDThh:mm:ss panic vpxd[34811] [Originator@6876 sub=Default]
  
  -->
     --> Panic: Memory exceeds hard limit. Panic

• /var/log/vmware/vmon/vmon.log gives below messages:
  

  YYYY-MM-DDThh:mm:ss In(05) host-<id> <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0
  YYYY-MM-DDThh:mm:ss In(05) host-<id> <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0

  
  
  

Environment

VMWare vCenter Server with External Backup solution

Cause

The crash was identified as being caused by the backup client utilizing a deprecated API (since vSphere API 4.1).

Resolution

Permanent Fix:

Work with your backup vendor to transition from the deprecated RetrieveProperties API/ RetrieveContents API to RetrievePropertiesEx / RetrieveContentsEx (utilizing ContinueRetrievePropertiesEx as necessary).

Workarounds:

• Reduce Concurrent Backups:

  • Split large backup jobs into smaller ones.

  • Reduce the number of virtual machines included in each backup schedule.

• Increase vmware-vpxd heap Memory:

  • Allocate additional memory to the VCSA to accommodate all incoming requests.

  • Reference KB 320871 for instructions on manually increasing heap memory on vCenter.

Note: An increase in the number of backup clients can lead to memory exhaustion, potentially reintroducing the issue.