vmware-vpxd service crash with core dump "core.vpxd-worker.xxxxxxx" on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• You will observe core dumps generated for each crash in the /var/core  core.vpxd-worker.xxxxxxx.

• /var/log/vmware/vpxd/vpxd.log will contains snippets similar to below,

YYYY-MM-DDThh:mm:ss error vpxd[34811] [Originator@6876 sub=Memory checker] Current value xxxxxxx exceeds hard limit xxxxxxx. Shutting down process.
YYYY-MM-DDThh:mm:ss panic vpxd[34811] [Originator@6876 sub=Default]

-->
   --> Panic: Memory exceeds hard limit. Panic

• /var/log/vmware/vmon/vmon.log

YYYY-MM-DDThh:mm:ss In(05) host-<id> <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0
YYYY-MM-DDThh:mm:ss In(05) host-<id> <vpxd> Service is dumping core. Coredump count 0. CurrReq: 0

Environment

VMWare vCenter Server with External Backup solution

Cause

The crash was identified as being caused by the backup client utilizing a deprecated API (since vSphere API 4.1).

 

Resolution

Permanent Fix:

Work with your backup vendor to transition from the deprecated RetrieveProperties API/ RetrieveContents API to RetrievePropertiesEx / RetrieveContentsEx (utilizing ContinueRetrievePropertiesEx as necessary).

Workarounds:

• Reduce Concurrent Backups:

  • Split large backup jobs into smaller ones.

  • Reduce the number of virtual machines included in each backup schedule.

• Increase vmware-vpxd heap Memory:

  • Allocate additional memory to the VCSA to accommodate all incoming requests.

  • Reference KB 320871 for instructions on manually increasing heap memory on vCenter.

Note: An increase in the number of backup clients can lead to memory exhaustion, potentially reintroducing the issue.