How to retrieve logs from an unresponsive VM
Article ID: 293388
Updated On:
Products
Operations Manager
Issue/Introduction
When a VM becomes unresponsive, BOSH commands such as bosh ssh and bosh logs are not available for retrieving the logs.
Resolution
IMPORTANT: This procedure can only be performed for the ephemeral disk. It can not be performed for the persistent disk.
It is possible to mount the disk that contains the logs onto a new VM and retrieve the logs this way. To do this follow the process outlined below.
Part 1: Create a rescue VM
1. Create a new Ubuntu VM (e.g., Ubuntu 20.04 or 22.04) in your IaaS. This new instance will serve as your "rescue VM" for the remainder of these steps. After creating it, follow the procedure in Part 2
Part 2: How to Mount the disk from the unresponsive VM
1. bosh update-resurrection off
2. Identify the VM ID for the rescue VM and also for the failed/unresponsive VM.
3. Connect to your IaaS Provider and locate the unresponsive VM.
4. Shutdown the VM gracefully.
5. Identify the ephemeral disk and its location.
6. Detach the disk without deleting the file.
7. Locate the rescue VM and attach the ephemeral disk file.
8. Reboot the rescue VM.
9. SSH into the rescue VM an run the following commands- sudo -i
- df -h ### confirm mounted volumes
- fdisk -l ### confirm available volumes after restart and identify the newly added volume (in this case, is sdc2)
- mkdir /tmp/sdc2
- mount /dev/sdc2 /tmp/sdc2
- cd /tmp/sdc2
You can now search for the logs and analyze the problem.
5. When finished, please delete the rescue Ubuntu VM that was created in part 1.
5. When finished, please delete the rescue Ubuntu VM that was created in part 1.
Note: The article covers the steps needed to retrieve logs from an unresponsive VM for vSphere. Please refer to the documentation for other IaaS providers for additional information on how to perform these steps.
Additional Information
Feedback
Yes
No
Powered by
