Understanding Application Log Limit Changes Starting in TAS v3.0

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This KB article aims to raise awareness on the changes to the log limiting feature starting in Tanzu Application Service (TAS) v3.0. 

Prior to TAS v3.0, the log limiting feature was configured on a global log-lines per second basis. For example, if platform operators wanted to limit the log throughput for noisy applications, they could implement a log-lines per second limit in the foundation. This feature gave platform operators the ability to relieve logging systems and downstream log-ingestion systems by limiting the amount of logs being sent through the platform. While this feature is very useful there are a few limitations with this approach:

• Every application is uniquely different and has different logging needs. 
• Log lines vary in size.

Given these limitations a new log limiting feature has been added starting in TAS v3.0 to accommodate non-global configurations measuring in bytes per second instead of log-lines per second. The deprecating global log-lines per second feature is still available in TAS v3.0 via the property max_log_lines_per_second however it also underwent some changes. Previously, logs that exceeded the log rate limit were buffered and released at the configured log rate for flow control. Starting in TAS v3.0+ the logs that exceed the log rate are now dropped.

The global log-lines per second feature was deprecated starting in TAS 4.0+, however it has been reintroduced as an available option in later versions. Please see the last section of this KB Deprecated Feature Reintroduced for more information.

More information about log rate limiting can be found in the official documentation.

Environment

Product Version: 3.0

Resolution

Starting in TAS v3.0+ every application in the foundation will have a log rate limit in bytes configured whether it's manually specified or inheriting a default value. The new TAS property cloud_controller_default_log_rate_limit_app sets the default value for apps to inherit if the limit is not manually specified. This property defaults to 16834 bytes and is configurable.

Upon an upgrade to TAS v3.0+, existing apps will be migrated with their log limits set to the value of -1 (unlimited). Setting the log limit to -1 allows existing apps to continue logging as they were before the upgrade however new app deploys will inherit the default value for the log rate limit if one is not specified. To elaborate on this concept consider the following scenario:

1. TAS is upgraded to v3.0+
2. App1 migrated up with value -1 for the log limit.
3. A blue/green deployment for App1 takes place, and no log limit is specified.

In this scenario, the newly deployed App1 will have a new guid and will be viewed as a new application to the platform. Since there was no log limit specified in the deploy, it will inherit 16834 from the default setting as its log limit. This may be undesired if App1 is unexpectedly limited on log throughput. App1 may see the following log indicating that the limit was exceeded:

2023-07-21T13:02:42.55-0400 [APP/PROC/WEB/0] OUT app instance exceeded log rate limit (16834 bytes/sec)

To quickly allow App1's logging to be unblocked from the limit we could use the cf CLI to scale the log limit: 

cf scale App1 -l -1

Once the log limit is updated  to -1 (unlimited) a restart the application is necessary. This unblocks App1's log limit and allows logs to start flowing again until a proper limit can be discovered and set.

The new log limiting feature in TAS v3.0+ is optimized for platform operators to have more granular control on platform logging activity however being aware of the feature's configuration options and behavior is important for a smooth experience.

Developers will have the ability to:

• Specify the log rate limit in the app manifest. More details about this manifest attribute can be found here.
• Use a compatible cf CLI version (v8.5+) and set the log rate limit via cf scale <APP-NAME> -l.

Platform Operators will have the ability to:

• Configure a maximum bytes of logs allowed per second per app at the org and space level via quotas. For more information on creating and modifying quota plans please see this documentation.
• Set a default bytes per second limit for applications to inherit if a limit is not specified in the app's manifest. See item 8 in the Configuring App Developer Controls docs.
• Configure app log rate limits for individual apps via cf scale <APP-NAME> -l.

When updating a log limit for an application, the application will need to be restarted before the new limit will take effect.

Each application instance will have a GAUGE metric emitted indicating its log_rate. For example:

2023-07-19T17:03:56.35-0400 [logger-app/4] GAUGE log_rate:8196.200000 B/s

indicates that instance 4 of logger-app is hitting a log_rate of 8196 bytes per second.

The log_rate metric may be helpful when determining log limit settings for a foundation. Every foundation is uniquely different and may have different requirements when strategizing the defaults to configure. An example strategy may look like the following:

1. Upgrade foundation to TAS v3.0 which will cause all apps to migrate with unlimited log limits.
2. Monitor the log_rate metrics in the foundation to get an idea of logging throughput needs. This area of investigation would include identifying noisy applications and validating if that app's logging throughput is necessary. 
3. Set a good foundation default limit for apps to inherit based on the findings from evaluating log_rate metric data. 
4. For any orgs or spaces that have unnecessarily noisy applications, configure a quota specifying the maximum allowed amount of log bytes per second and apply to that org or space. 
5. Manually update log limits on any remaining applications not covered under the previous steps.

Deprecated Feature Reintroduced

The global log-lines per second feature was removed from the following versions of TAS:

• v4.0.0 - v4.0.19
• v5.0.0 - v5.0.9

The global log-lines per second feature was reintroduced starting in the following versions of TAS:

• v4.0.20+
• v5.0.10+