Intermittently vpxd service is getting crashed with core dump.

Products

• VMware vCenter Server

Issue/Introduction

• vpxd service crashes intermittently while powering on vCLS VM on vSAN cluster. 
• In the vSphere UI, we observe below screenshot. 

Environment

VMware vCenter Server 7.X
VMware vCenter Server 8.X

 

Cause

• vCLS VMs have these 2 properties to differentiate between normal VM and system managed VM. 
  • ManagedByInfo.extensionKey property is equal to "com.vmware.vim.eam"
  • ManagedByInfo.type property is equal to "cluster-agent"
• When these parameters are deleted or unset from vCLS VMs, vCenter will treat them as normal VM. 
• Use below URL to check the config.managedBy properties for the VMs. 

  • https://{vcIP}/mob/?moid={vm-moId}&doPath=config.managedBy

• Due to an unknown reason, the vSAN cluster was shutdown. When we restarted the vSAN cluster, vsanvcmtd requested EAM to enable vCLS VMs., then EAM attempted to power on the vCLS-XXXXXX VM.
• While powering On the vCLS VM vpxd service crashes with core dump. Upon verifying the config.managedBy properties, the above parameters are missing for vCLS VMs. 

Resolution

• Enable retreat mode to remove the existing vCLS VMs under affected cluster. 
  • vSphere 7.0 U3o/8.0 U2 and Later
    1. Log into vCenter's HTML5 client
    2. In Hosts and Clusters inventory, select a cluster.
    3. Click on the Configure tab.
    4. Under vSphere Cluster Services, select General.
    5. In the top right, click on EDIT VCLS MODE.
    6. In the Edit vCLS Mode pop up window, click on the second radio option Retreat Mode.
    7. Click OK.
• Disable Retreat mode to create vCLS VMs. 
  • vSphere 7.0 U3o/8.0 U2 and Later
    1. Log into vCenter's HTML5 client
    2. In Hosts and Clusters inventory, select a cluster.
    3. Click on the Configure tab.
    4. Under vSphere Cluster Services, select General.
    5. In the top right, click on EDIT VCLS MODE.
    6. In the Edit vCLS Mode pop up window, click on the second radio option System Managed.
    7. Click OK.