App Autoscaler can cause high CPU usage on the PAS MySQL cluster in large environments

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

If you have large foundations and with many applications registered to the Autoscaler, you will notice increased CPU consumption on your MySQL server.

This has been observed in foundations where a thousand or more applications are registered with Autoscaler. This is not an exact marker. You may see it occur with fewer or more applications registered with Autoscaler, depending on how much CPU is available to your MySQL cluster.

Environment

Cause

The issues is caused by the Autoscaler sending notifications for scaling events. Over time the table that backs this functionality grows too large for the existing queries to be efficient. The queries will continue to run, but they will take longer to execute and consume more CPU to complete.

Resolution

There are two workarounds for this issue:

1. Increase the resources for your MySQL cluster. If you add additional VCPUs to the MySQL servers in your cluster, this will provide additional capacity and can mitigate this issue.

• You can do this by selecting a larger VM Type on the PAS Resource Configuration screen in Ops Manager.

2. Disable notifications. While this resolves the issue, the trade-off is that you will no longer receive email notifications for scaling events.

To disable notifications there are two options.

a. If you are running a version of TAS prior to 2.5, you can do this manually.

• Customers can disable the notifications by setting the NOTIFICATIONS_HOST environment variable to an invalid hostname.

• For example, if NOTIFICATIONS_HOST=notifications.example.com, change it to NOTIFICATIONS_HOST=invalid-notifications.example.com. Follow the below steps to make the change:

1. Run: cf t -o system -s autoscaling

2. Run: cf setenv autoscaler NOTIFICATION_HOST invalidhostname.example.com

3. Run: cf restage autoscaler

b. If you are running TAS 2.5+, go to the TAS tile in Ops Manager. Click on the App Autoscaler section on the left and uncheck the box "Enable email notifications". Save and apply changes.