vmware-vpxd crash after incomplete storage vMotion leaves duplicate, inaccessible VM in vCenter inventory

Products

• VMware vCenter Server 8.0

Issue/Introduction

• VPXD crash occurred shortly after an incomplete Storage vMotion left a partially registered, inaccessible duplicate VM in the vCenter inventory. 
• The vmware-vpxd service is crashing intermittently, producing core dump files in the directory /var/core/.
• The service status on the vCenter Server shows vmware-vpxd service in stopped state.

# service-control --status --all
   Running:
   applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-wsla-broker vlom vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement
   vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver
   vmware-rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd-svcs
   vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
   stopped:
   vmcam vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha vmware-vpxd

• Log file /var/log/vmware/vmon/vmon.log in vCenter shows below entries

YYYY-MM-DDTHH:MM:SS In(05) host-2582 <vpxd> Service is dumping core. Coredump count 9. CurrReq: 0
YYYY-MM-DDTHH:MM:SS In(05) host-2582 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vpxd,UNHEALTHY,HEALTHY,1
YYYY-MM-DDTHH:MM:SS Wa(03) host-2582 <vpxd> Service exited. Exit code 1
YYYY-MM-DDTHH:MM:SS Wa(03) host-2582 <vpxd> Service exited unexpectedly. Crash count 9. Taking configured recovery action.
YYYY-MM-DDTHH:MM:SS Wa(03) host-2582 Failed to publish health status change.

• No panic or crash traces are seen in vpxd logs located at /var/log/vmware/vpxd.
• Starting the vpxd service from vCenter shell by running the following command will only keep the vpxd service running for a short period of time.

  service-control --start vmware-vpxd

Environment

• vCenter 8.x

Cause

• The VPXD service crash was caused by the presence of an duplicate VM in the vCenter database.

Resolution

1. Contact Broadcom Technical Support, attaching vCenter logs 
2. Confirm the duplicate VM from the core dump analysis is unused, and then remove the duplicate VM from the vCenter inventory.
3. Restart all vCenter services
   
   service-control --stop --all
service-control --start --all

Note : Attempt to remove the VM from the vCenter inventory. If this operation fails, the VM must be removed directly from the vCenter database - Manually removing a stale VM from the vCenter Server vpostgres database

Additional Information

Ensure all collected core dump files are deleted after analysis is complete, as the /storage/core volume will be fully occupied by these core dump files