How to retrieve logs from an unresponsive VM
search cancel

How to retrieve logs from an unresponsive VM

book

Article ID: 293388

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

BOSH commands such as bosh ssh and bosh logs fail with "Timed out sending 'get_state'" or "Timed out pinging VM". This often occurs during heavy I/O operations like backups or after infrastructure restarts.
 
 

Environment

  • VMware Tanzu Application Service (TAS)
  • VMware Tanzu Kubernetes Grid Integrated (TKGI)
  • vSphere

Resolution

IMPORTANT: This procedure can only be performed for the ephemeral disk. It can not be performed for the persistent disk. 

It is possible to mount the disk that contains the logs onto a new VM and retrieve the logs this way. To do this follow the process outlined below.


Part 1: Create a rescue VM 

1. Create a new Ubuntu VM (e.g., Ubuntu 20.04 or 22.04) in the IaaS to serve as a rescue VM.

 

Part 2: How to Mount the disk from the unresponsive VM

  1. Run bosh update-resurrection off.
  2. Identify the VM ID for the rescue VM and also for the failed/unresponsive VM.
  3. Connect to your IaaS Provider and locate the unresponsive VM.
  4. Shutdown the VM gracefully.
  5. Identify the ephemeral disk and its location.
  6. Detach the disk without deleting the file.
  7. Locate the rescue VM and attach the ephemeral disk file.
  8. Reboot the rescue VM.
  9. SSH into the rescue VM an run the following commands
    sudo -i 
df -h ### confirm mounted volumes
fdisk -l ### confirm available volumes after restart and identify the newly added volume (in this case, is sdc2)
mkdir /tmp/sdc2
mount /dev/sdc2 /tmp/sdc2
cd /tmp/sdc2
  10. Retrieve logs from /var/vcap/sys/log to analyze the problem. 
  11. Delete the rescue Ubuntu VM when finished.
 
Note: The article covers the steps needed to retrieve logs from an unresponsive VM for vSphere. Please refer to the documentation for other IaaS providers for additional information on how to perform these steps. 

 



Additional Information

Links to other IaaS: 

 

Powered by Wolken Software