How to retrieve logs from an unresponsive VM
search cancel

How to retrieve logs from an unresponsive VM

book

Article ID: 293388

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

When a VM becomes unresponsive, BOSH commands such as bosh ssh and bosh logs are not available for retrieving the logs.

Resolution

IMPORTANT: This procedure can only be performed for the ephemeral disk. It can not be performed for the persistent disk. 

It is possible to mount the disk that contains the logs onto a new VM and retrieve the logs this way. To do this follow the process outlined below.


Part 1: Create a VM 

1. Create an Ubuntu 16.04 VM in your IaaS and then follow the procedure in part 2 to mount the disk from the unresponsive VM.

 

Part 2: How to Mount the disk from the unresponsive VM


1. bosh update-resurrection off

2. Identify the VM ID for the rescue deployment and also for the failed/unresponsive VM. 

3. Connect to your IaaS Provider and locate the unresponsive VM. 
 
4. The next steps can be performed from within the deployment. 
  • SSH into the rescue VM
  • sudo -i 
  • df -h ### confirm mounted volumes
  • fdisk -l ### confirm available volumes after restart and identify the newly added volume (in this case, is sdc2)
  • mkdir /tmp/sdc2
  • mount /dev/sdc2/tmp/sdc2
  • cd /tmp/sdc2
You can now search for the logs and analyze the problem. 

5. When finished, please delete the one-off Ubuntu VM that was created in part 1. 
 
Note: The article covers the steps needed to retrieve logs from an unresponsive VM for vSphere. Please refer to the documentation for other IaaS providers for additional information on how to perform these steps. 
  • Shutdown the VM gracefully.
  • Identify the ephemeral disk and it's location in the datastore. 
  • Detach the disk without deleting the file. 
  • Locate the rescue VM and attach the ephemeral disk file from the previous step. 
  • Reboot the rescue VM. 
Links to other IaaS: 



Powered by Wolken Software