Determining which user removed or deleted a virtual machine

Products

• VMware vCenter Server

Issue/Introduction

This article provides steps to determine which user deleted a virtual machine or removed a virtual machine from the vCenter Server inventory.

• A virtual machine is missing from the vCenter Server inventory.
• VM records are no longer visible in the vSphere Client.

Environment

VMware vCenter Server 

Resolution

To identify the user who removed or deleted a virtual machine, review the vCenter Server events and logs corresponding to the time of the deletion. The investigation can be performed from either the ESXi host logs or the vCenter Server logs.

Note: Log files in vCenter Server are subject to rotation. If the deletion event occurred outside the current retention period, the logs may no longer be available. It is recommended to investigate deletion events as soon as they are detected.

Identify via vCenter Server Logs

Use this method for centralized investigation of vCenter-level events.

1. Login to vCenter via SSH as the root user.
2. Execute the following command to query system events:
   journalctl | grep -i "VmRemovedEvent"
   
   Log Snippet:
   YYYY MM DD hh:mm:ss <vc fqdn> vpxd[##]: Event [##] [1-1] [YYYY MM DD hh:mm:ss] [vim.event.VmRemovedEvent] [info] [domain\user] [datacenter-#] [##] [Removed bootstrap-vm-<vm name> on <esxi fqdn> from datacenter-##]

Identify via ESXi Host Logs

Use this method if the VM removal occurred on a specific ESXi host.
Note: If the VM was deleted via vCenter Server, the ESXi log will show the user as vpxuser. To find the specific Active Directory or SSO user, you must check the vCenter Server logs (Method 2).

1. Connect to the ESXi host on which the virtual machine resided prior to deletion via SSH as root user.
2. Navigate to the log directory using: cd /var/run/log/
3. Execute the following command to search for deletion events: grep -i "Destroy VM called" hostd*
   
   Log snippet:
   YYYY-MM-DD hh:mm:ss In(166) Hostd[#####]: [Originator@## sub=Vmsvc.vm:/vmfs/volumes/datastore/VM Name/VM name.vmx opID=#### sid=#### user=vpxuser:domain\Administrator] Destroy VM called
YYYY-MM-DD hh:mm:ss In(166) Hostd[#####]: [Originator@## sub=Vmsvc.vm:/vmfs/volumes/datastore/VM Name/VM name.vmx opID=#### sid=#### user=vpxuser:domain\Administrator] State Transition (VM_STATE_OFF -> VM_STATE_DELETING)
YYYY-MM-DD hh:mm:ss In(166) Hostd[#####]: [Originator@## sub=Vmsvc.vm:/vmfs/volumes/datastore/VM Name/VM name.vmx opID=#### sid=#### user=vpxuser:domain\Administrator] Destroy VM complete

Trace Source IP for API/Automated Deletions

For deletions triggered by service accounts, API calls, or automated tools, you must correlate the session ID to a network source IP to reach a conclusive finding.

Additional Information

Identifying the user who deleted the virtual machine