how to collect the logs for OpsManager Bosh director and system logs for Bosh VM managed

Products

• VMware
• VMware vSphere with Tanzu

Issue/Introduction

It also includes some additional logs like system logs and Bosh agent that could be useful for troubleshooting.

Symptoms:
This KB explain how to collect the logs for OpsManager, Bosh director and any Bosh VM managed.

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x
Tanzu Kubernetes Grid Integrated Edition 1.1.14.1

Resolution

1. Collect the OpsManager bundle
 From OpsManager GUI -> click on Support button right bottom on the main page


2. Collect the Bosh director logs
 You can use the Bosh director bbr SSH key for transferring the logs from the Bosh Director VM to your jumpbox.
There are 3 repositories in the Bosh director VM that contain the services logs monitored by monit, the system logs (kern.log, audit.log ,sysstat ...) and the Bosh log agents.

2.1. Retrieve the private SSH key from the OpsManager Bosh tile -> Credentials tab -> Bbr Ssh Credentials -> private_key_pem value and excluding the leading and trailing double quotes, paste it into a file called bbrkey_director.pem

2.2. Format the file bbrkey_director.pem and change the permissions to read only:

echo -e $(cat bbrkey_director.pem) > bbrkey_director.pem
chmod 400 bbrkey_director.pem

2.3. SSH into the Bosh director VM with the bbr user and key:

ssh -i bbrkey_director.pem -o StrictHostKeyChecking=no bbr@DIRECTOR_IP

2.4. Create 3 archives for the following repositories from the Bosh director VM:

sudo tar -czvf /var/vcap/data/varsyslog_director.tgz /var/vcap/sys/log && sudo tar -czvf /var/vcap/data/varbosh_director.tgz /var/vcap/bosh && sudo tar -czvf /var/vcap/data/varlog_director.tgz /var/log

2.5. Collect the 3 archives using SCP from your jumpbox:

scp -i bbrkey_director.pem -o StrictHostKeyChecking=no bbr@DIRECTOR_IP:/var/vcap/data/*director.tgz .

2.6. Clean up the 3 archives using SSH from your jumpbox:

ssh -i bbrkey_director.pem -o StrictHostKeyChecking=no bbr@DIRECTOR_IP 'sudo rm /var/vcap/data/*director.tgz'

2.7 Collect the Bosh director tasks and events history for all users for a specific deployment:

bosh -d DEPLOYMENT events > bosh_events.txt
bosh -d DEPLOYMENT tasks -r=20000 --all > bosh_tasks_20000.txt

2.8 For investigating a specific Bosh task, capture the debug log:

bosh task ID --debug > bosh_task_ID_debug.txt

3. Collect the Bosh VM managed that include system logs and Bosh agent
 The command bosh -d DEPLOYMENT logs INSTANCE only collects the logs located in /var/vcap/sys/log folder.
To collect the services logs monitored by monit, the system logs (kern.log, audit.log ,sysstat ...) and the Bosh log agents, you can run the following commands against a single instance or a group instance (ie: master, worker, router, mysql...)

bosh -d DEPLOYMENT ssh master -c "sudo tar -czvf /var/vcap/sys/log/varlog.tgz /var/log && sudo tar -czvf /var/vcap/sys/log/varbosh.tgz /var/vcap/bosh/log"

bosh -d DEPLOYMENT logs master

bosh -d DEPLOYMENT ssh master -c "sudo rm /var/vcap/sys/log/varlog.tgz /var/vcap/sys/log/varbosh.tgz"

4. How to generate additional logs (optional)
 If you would like to generate additional logs that are not stored in Bosh VM managed, please follow this KB https://kb.vmware.com/s/article/90321