vCenter vpxd service crashes unexpectedly during backup operations due to VM with missing instanceuuid.

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter Server (vpxd) service crashes repeatedly, generating core.vpxd-worker dump files.
• The Virtual machines are missing instanceuuid parameter when validated.
• The failure is triggered when 3rd party backup attempts to interact with Virtual Machines on the vCenter server.

Environment

vCenter server 8.0U3.

Cause

The vpxd service crashes when the backup software attempts to query a virtual machine that is missing its UUID.

Resolution

• Broadcom Engineering is aware of this issue and is working on a fix to ensure the vpxd service handles missing instanceuuid values in the future.
  
  

To resolve this issue update the missing instanceuuid for the affected VM(s) using Power CLI:

• To Get BIOS and Instance UUIDs for all VMs
  
  Get-VM | Select-Object Name, 
    @{N="BIOS_UUID"; E={$_.ExtensionData.Config.Uuid}}, 
    @{N="Instance_UUID"; E={$_.ExtensionData.Config.InstanceUuid}}
  
  
•  To Get VMs with null Instance UUIDs 
  
  Get-VM | Where-Object {$_.ExtensionData.Config.InstanceUuid -eq $null} | Select-Object Name
  
  
• Steps to update uuid for the VM

1. Generate a new GUID
$newGuid = (New-Guid).ToString()
2. Specify the VM name that has missing UUID
$vmName = "VMName"
3. Create the spec and apply
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.InstanceUuid = $newGuid
$vm = Get-VM -Name $vmName
$vm.ExtensionData.ReconfigVM_Task($spec)

• To Verify newuuid for the VMname

(Get-VM -Name $vmName).ExtensionData.Config.InstanceUuid

• To list uuid for the VMName

Get-VM -Name $vmName | Select-Object Name, 
    @{N="BIOS_UUID"; E={$_.ExtensionData.Config.Uuid}}, 
    @{N="Instance_UUID"; E={$_.ExtensionData.Config.InstanceUuid}}

This should help the vCenter server to be stable and accessible.

Additional Information

Manually updating the UUID in the vmx file

VMware PowerCLI User's Guide