Artifacts to upload to CloudCache (PCC) tickets

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article refers to Artifacts to upload to CloudCache (PCC) tickets

Environment

Resolution

Logs required for issues related to upgrade and install failures

• PCC broker logs
  • This can be obtained from OpsManger->Pivotal Cloud Cache Tile -> Status -> Logs. 
• Debug logs for the failed task
  •  This can be obtained by running bosh -e <ENV> task <TASK_ID> —debug > <TASK_ID>_debug.log. How to get the <TASK_ID> is described in the last section.
• Failed PCC Service instance logs
  • This can be obtained by running bosh -e <ENV> -d <SERVICE_INSTANCE_ID> logs. How to get <SERVICE_INSTANCE_ID> is described in the last section.

Logs/artifacts required for all other issues

• PCC Service instance logs by running bosh -e <ENV> -d <SERVICE_INSTANCE_ID> logs. How to get <SERVICE_INSTANCE_ID> is described in the last section.
• GemFire statistics and log files which can be obtained as described in the PCC docs. 
  • If the size of log/stats files are big gfsh>export logs command will give say so instead of downloading. gfsh>export logs command has various options that you can use to reduce the size of the zip file being download. If you know the timeframe of the problem you could use start-time and end-time options. Details of the command can be in GemFire docs
• If gfsh is not accessible: We can get the logs by ssh`ing into each VM of the service instance and get the logs from /var/vcap/sys/log   and stats from /var/vcap/store/  for GemFire processes.
• Thread dumps: We can get the thread dumps by ssh`ing into each VM of the service instance and get the thread dump from /var/vcap/packages/jdkXX/bin by running [jcmd/jps/jstack] pid to get thread dump. 
   

How to get <TASK_ID> and <SERVICE_INSTANCE_ID>:

When apply changes fail, Ops Manager log will tell you the taskID which failed along with the service-instance-guid. The log snippet in opsmanager log will look something like the below. One can get the task ID and service instance ID from there.

Last Operation 
Status: create failed 
Message: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p-cloudcache, service-instance-guid: sdf8c3569-####-####-####-hg5303gdbc651, broker-request-id: 0b4eba99-####-####-####-da11914b2f55, task-id: 45526, operation: create 
Started: 2018-03-16T05:31:28Z
Updated: 2018-03-16T05:35:38Z

In this example TASK_ID is 45526 and the SERVICE_INSTANCE_ID is sdf8c3569-####-####-####-hg5303gdbc651.