Cloud controller requests.outstanding metric is stuck

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Using monitoring tools such as healthwatch you noticed that cloud controller cc.requests.outstanding metric suddenly becomes stuck. Here is a sample graph, notice the metric did not come come to zero around 10/15 

Environment

Product Version: 2.11

Resolution

There is a known behaviour with cloud_controller when stopped/killed in the middle of handling requests this metric would get stuck. 

To resolve this:

1. ssh into one of the cloud controller vms 

   bosh -d <tas-deployment> ssh cloud_controller/0

    
2. stop cloud_controller_ng job 

   monit stop cloud_controller_ng

3. stop statsd_injector job 

   monit stop statsd_injector

4. check if the jobs were stopped 

   monit summary

5. start cloud_controller_ng job

    monit start cloud_controller_ng

6. start statsd_injector 

   monit start statsd_injector

   
    
7. check if the jobs were started successfully 

   monit summary

8. Repeat Steps 1 to 7 to the other cloud controller vms