App Autoscaler fails to auto scale apps which generate high volume of logs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

On the same TAS platform, App Autoscaler can successfully auto scale most apps but fails to scale apps which are generating very high volumes of logs. 

Resolution

App Autoscaler depends on metrics from Log Cache to make scaling decisions. As described in the diagram below, Autoscaler worker fetches app metrics from Log Cache at the default of a 120s interval. The interval can be configured at TAS > Settings > Configure App Autoscaler > Metric collection interval at 60s to the 3600s range. 

 

Log Cache is shared by all apps running on the platform, for each app (not instance), Log Cache allocates a buffer with a default 100K message length. This is configurable at TAS > Settings > Advance Feature. App logs and metrics share the same buffer, thus when an app is generating logs too fast, the metrics data Cache Duration is even less than the interval in which Autoscaler fetches the metrics. Autoscaler won't be able to make scaling decision without metrics. To identify if any apps are being impacted, please refer to the following KB:
 

• Identify noisy applications with Log-Cache in Tanzu Application Service for VMs (TAS for VMs) 2.2 and above.

If the "Cache Duration" is less than 120s, the app autoscaling feature will be impacted. 

 

To resolve the problem, the following is recommended:

• Reduce unnecessary app logs to make the "Cache Duration" longer at least than 120s. 
• Increase Log Cache buffer at "TAS > Settings > Advanced Features > Maximum number of envelopes stored in Log Cache per source"