Example Case: A virtual machine was powered off by the user
[email protected] on 07/15/2016. The organization has several users that have access to this account and need to determine the IP address of the virtual machine power off task.
When a user connects to vCenter Server (vpxd) through API's or the vSphere client, a session ID is created. vCenter Server logs the username and IP address for each session created. While reviewing these logs, you see entries similar to:
vpxd.log
2016-07-15T13:47:32.165-04:00 [03632 info 'commonvpxLro' opID=632c2fd2] [VpxLRO] -- BEGIN task-445 -- vm-41 -- vim.VirtualMachine.powerOff -- 74656bc4-####-####-####-########375(526d385a-####-####-####-########0bd)
The vpxd.log contains a virtual machine power off task that was created with a session ID of 74656bc4-####-####-####-########375.
vpxd-profiler.log
/SessionStats/SessionPool/Session/Id='74656bc4-####-####-####-########375'/Username='DOMAIN.LOCAL\AUDIT'/ClientIP='192.168.2.3'
The vpxd-profiler.log shows session ID 74656bc4-####-####-####-########375 was created by user DOMAIN.LOCAL/AUDIT from the IP address 192.168.2.3.
To determine the correct log message that corresponds to the desired task in the vpxd.log. To try and narrow down the user:
- Find the exact time that the task took place from the tasks tab for the object of interest.
For example:
If the task of interest is related to a virtual machine, perform the following steps in the vSphere Web Client:
- Select the virtual machine in the left-hand pane of the vSphere Web Client.
- Select Monitor.
- Select Tasks.
- Search for the event requiring review.
- Once the desired task is found, it will show the start time of the task. This is helpful as each log message in vpxd.log starts with the time stamp.
Note: In the example the time stamp is 2016-07-15T13:47:32.165-04:00.
- The log sequence for tasks in vpxd.log starts with BEGIN.
- The log message produced by the task frequently has a relevant word linked to the task of interest.
Note: In the example the log message contains the word vim.VirtualMachine.powerOff, which indicates a virtual machine power off task.
- Use the VMID to confirm that the log message corresponds to the correct virtual machine. The VMID can be mapped back to the name of the virtual machine using the commands in Collecting information about tasks in VMware ESXi/ESX (1013003).
Note: In the example the VMID for the virtual machine power off task is vm-41.
- Optional:Reproduce the desired task in a lab environment. It is much easier to ensure the correct log message is located after reproducing the task and viewing the logs immediately after.
Example workflow:
- Find the desired task in the vSphere Web Client to locate the username and the start time.
- Open the vpxd.log and review the log section near the start time of the task.
- Look for a log message with the word BEGIN around the start time.
- Search for any task description that seems relevant. If the task is related to a virtual machine, then confirm that the VMID in a suspected log message matches to the correct virtual machine.
- Reproduce the task in a lab environment and confirm that the same log messages are created.
- Open the vpxd-profiler.log and find the originating IP address that corresponds to the session ID found in the vpxd.log.