How to troubleshoot App Autoscaler issues in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

App Autoscaler on VMware Tanzu Platform is composed of two apps under the system Org, autoscaling Space and database autoscale.

• autoscale: includes both broker and worker at route autoscale.<SYSTEM_DOMAIN>. The worker depends on TAS log-cache to receive app metrics. It scales apps according to received metrics and the configured rules or instances limit.

• autoscale-api: API endpoint at route autoscale.<SYSTEM_DOMAIN>/api/v2. It serves the following requests: enable/disable autoscaling, create/delete/update rules, view events, create/delete instances limit. 

• autoscale database: when using internal database for Tanzu Application Service, the database can be accessed with steps:

  1. $ bosh -d DEPLOYMENT ssh mysql/0

  2. $ sudo -i

  3. # mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf

  4. mysql> use autoscale; 

This article demonstrate general troubleshooting steps when hitting an issue with App Autoscaler in Tanzu Application Service for VMs (TAS for VMs).

Resolution

Binding or unbinding issues

When hitting problem with service binding or unbinding, please check the broker app autoscale logs. 

Apps Manager configuration issues

When hitting problems with enabling and disabling Apps Manager or configuring rules/limits via Apps Manager and the cf CLI autoscale plugin. The autoscale-api is the API endpoint handling those requests, please check the API's logs to see if it receives and responds with the correct HTTP status code. For more information on using the cf CLI, refer to Using the App Autoscaler CLI.

Apps Manager is placed into an enable or disable state from autoscale-api via the search-server, while the plugin gets the state from autoscale-api directly. In case the operation works well with plugin commands, but fails with Apps Manager web UI, the problem is mostly like introduced by search-server component. This component is under the system org / system space, please check its logs if there are any errors. 

The Autoscale app is also the worker in the case the service can be bound and the rules or limits can be successfully configured. However, if the app can't be auto-scaled, please turn on debug and collect the logs. The debug logs provide details how the worker is running in the background. 

1. cf set-env autoscale LOG_LEVEL debug
2. cf restart autoscale
3. cf logs autoscale

If the root cause can not be identified or resolved with these steps, please collect the relevant logs above and contact Tanzu Support.

Log Cache issues

In addition, autoscale (the worker) scales app instances based on metrics received from log-cache or RabbitMQ (queue depth). In case log-cache stops functioning for whatever reason, autoscale would not be able to scale app instances, no matter volatile the workload. To check  log-cache availability, please run the `cf app` command against any running apps. Confirm their CPU/memory/disk utilization is not zero - zero indicates that log-cache returns none metrics data. Another option is to `cf tail APP_NAME` with log-cache CLI plugin, zero envelopes indicate that log-cache is broken. 

In the case autoscale stops functioning due to log-cache issue, please review log-cache logs on doppler VMs and reach out to support with the logs if you need assistance.