Application auto-scaling fails with "Autoscaler did not receive any metrics for rabbitmq.policyload during the scaling window. Scaling down will be deferred until these metrics are available."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Autoscaler cannot receive metrics for the queue depth or queue length. You see auto-scaling events such as:

019-09-27T17:49:09Z Autoscaler did not receive any metrics for rabbitmq.policyload during the scaling window. Scaling down will be deferred until these metrics are available.

As of May, 2021, the Autoscaler app fetches queue depth information from the RabbitMQ server management API.

The Autoscaler app gets the http-management-api endpoint to make this request from the app's environment variable VCAP_SERVICES. The Autoscaler app obtains this VCAP_SERVICES through a cloud-controller-api call to get the app's env.

However, in RabbitMQ tile v1.13-current, there is a setting that allows the bound app's credentials to be stored in CredHub for on demand service instances.




When this setting is checked, the VCAP_SERVICES variable for the bound apps does not contain the http-management-api value and instead it looks like this:



In the case that this setting is checked, bound apps cannot scale based off of queue depth because when Autoscaler makes the API call to get the env variable value for the app, it is unable to get the http-management-api endpoint. Therefore, Autoscaler cannot make the request to get the queue depth.

Environment

Product Version: 2.5

Resolution

Workaround

To avoid this issue, follow one of the workarounds below: 

• Export custom metrics from your app to indicate the queue depth. Autoscaler is able to set rules off of custom metrics.
• Disable the feature to store service instance credentials in CredHub until Autoscaler is able to support this feature.