Troubleshooting bbsLRPsExtra metric increase

Products

• VMware Tanzu Application Service

Issue/Introduction

This article will details some methods for troubleshooting an unexpected increase in the TAS metric bbs.LRPsExtra.

A brief summary of this metric is:

Total number of LRP instances that are no longer desired but still have a BBS record. When Diego wants to add more apps, the BBS sends a request to the Auctioneer to spin up additional LRPs. LRPsExtra is the total number of LRP instances that are no longer desired but still have a BBS record.

This means that there are potentially records of LRPs remaining in the database that are no longer desired and should be deleted. More info on this metric can be found here, under "BBS time to handle requests":

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-platform-for-cloud-foundry/6-0/tpcf/monitoring-kpi.html

To confirm what your current value for this metric is, you can use the below cf nozzle command. 

cf nozzle -f ValueMetric | grep -e 'origin:"bbs"' | grep -i extra

The cf nozzle plugin needs to be installed from here:

https://github.com/cloudfoundry-community/firehose-plugin

Environment

All TAS envs.

Resolution

The main identifier of extra LRPs are entries in the diego database that are in a CLAIMED state, but have no actual application running behind them.

To identify these, first we check the TAS Diego DB:

• bosh ssh to the TAS MySQL VM
• sudo -i
• cat /var/vcap/jobs/pxc-mysql/config/mylogin.cnf
• using the password from above, run this command:
• mysql -u root -p<password> -D diego -e "SELECT * 
  FROM actual_lrps
  JOIN domains ON actual_lrps.domain = domains.domain
  LEFT JOIN desired_lrps ON actual_lrps.process_guid = desired_lrps.process_guid
  WHERE actual_lrps.presence = 0 AND desired_lrps.process_guid IS NULL" 

This should provide you with a list of LRPs in the claimed state. Ideally, this output should match the current value of the bbsLRPsExtra. 

The safest way to reduce the bbsLRPsExtra is to recreate the diego_cell that is supposed to be hosting the app instance. For that, take the VM instance ID from the 'cell_id' column in the output of the previous SQL query and run

bosh -d <cf-deployment> recreate diego_cell/<cell_id>

for each different diego_cell that is shown in the query output.

After recreating all listed diego_cells, metric bbs.LRPsExtra should get back to 0.