Collect NSX-T NCP logs in PKS environment

Products

• VMware NSX

Issue/Introduction

• Provide instructions to collect NCP logs with different options.
• This KB article describes how to collect NCP logs in PKS (1.2/ 1.3)  for NSX-T troubleshooting.

Environment

VMware NSX-T Data Center

VMware NSX

Resolution

• Get the deployment id of the kubernetes cluster and the instance id of the VM (master or worker))
  Login Opsmanager or other machine with bosh client and kubectl client installed 
  
  # bosh vms
  Using environment '172.18.1.41' as client 'ops_manager'
  
  Task 36563
  Task 36562
  Task 36563 done
  Task 36562 done
  
  Deployment 'pivotal-container-service-1ad5b09794ec7b43160d'
  
  Instance                                                        Process State  AZ  IPs          VM CID                                   VM Type  Active
pivotal-container-service/3069599d-ba6b-####-####-########004  running        AZ  172.18.1.42  vm-062fcfbc-962b-####-####-########80b  large    true
  
  1 vms
  
  Deployment 'service-instance_60c7bef1-a796-####-####-########707'
  
  Instance                                     Process State  AZ  IPs         VM CID                                   VM Type  Active
master/a8463a9d-8d06-####-####-########f82 running        AZ  172.30.0.2  vm-df8c41c0-9827-####-####-########04d medium   true
worker/9100545d-c5e2-####-####-########966 running        AZ  172.30.0.3  vm-d68ef49e-aa6d-####-####-########9c8 medium   true
  
  2 vms
  
  
• Generate NCP logs
  
  Option 1:  Generate service instance logs including NCP logs. 
  
  ~# bosh logs -d service-instance_60c7bef1-a796-####-####-########707
  Using environment '172.18.1.41' as client 'ops_manager'
  
  Using deployment 'service-instance_60c7bef1-a796-####-####-########707'
  
  Task 36690
  
  Task 36690 | 18:43:37 | Fetching logs for worker/9100545d-c5e2-####-####-########966 (0): Finding and packing log files
  Task 36690 | 18:43:37 | Fetching logs for master/a8463a9d-8d06-####-####-########f82 (0): Finding and packing log files
  Task 36690 | 18:43:42 | Fetching logs for worker/9100545d-c5e2-####-####-########966 (0): Finding and packing log files (00:00:05)
  Task 36690 | 18:43:58 | Fetching logs for master/a8463a9d-8d06-####-####-########f82 (0): Finding and packing log files (00:00:21)
  Task 36690 | 18:44:01 | Fetching group of logs: Packing log files together
  
  Task 36690 Started  Mon Apr  1 18:43:37 UTC 2019
  Task 36690 Finished Mon Apr  1 18:44:01 UTC 2019
  Task 36690 Duration 00:00:24
  Task 36690 done
  
  Downloading resource 'c3f46a3d-ba9b-####-####-########346' to '/root/service-instance_60c7bef1-a796-####-####-########707-########-######-42423186.tgz'...
  
  ############################################                75.82% 75.23 MB/s 0s
  Succeeded
  
  Option 2:  Generate NCP logs by logging in PKS master node:
  
  # bosh -d service-instance_60c7####-####-########001-cfa0aca85707 ssh master/a8463a9d-8d06-####-####-########f82
  Using environment '172.18.1.41' as client 'ops_manager'
  
  Using deployment 'service-instance_60c7bef1-a796-####-####-########707'
  
  Task 36712. Done
  Last login: Mon Apr  1 18:47:43 2019 from 172.18.1.181
  To run a command as administrator (user "root"), use "sudo <command>".
  See "man sudo_root" for details.
  
  master/########-####-####-####-########ef82:~$
  
  $ sudo su -
  master/########-####-####-####-########ef82:~#
  
  # tar -zcvf /tmp/var-log-ncp.tar.gz /var/log
  * Note this collect all logs including NCP
  
  For live review, NCP logs are under /var/vcap/sys/log/ncp folder in master node
  ncp.stdout.log
  ncp.stderr.log