log-cache memory utilization keeps growing up to 90%~

Products

• VMware Tanzu Application Service

Issue/Introduction

After upgrading to Tanzu Platform Elastic Application Runtime 10.3 release, it is observed that log-cache VMs memory utilization grow up to as high as 90%~, and very close to 100%. It causes some VMs slow and unresponsive. The memory utilization used to be as high as ~65% in previous releases. 

Environment

Tanzu Platform Elastic Application Runtime 10.3 release

Cause

This is caused by a breaking change in 10.3 release, log-cache property memory_limit_percent default value was increased from 50 to 85. This value control maximum heap size in log-cache process. Plus log-cache process non-heap and memory being used by other processes, the total memory utilization would be very close to 100%. 

Resolution

This has been raised up to engineering for review and fix.

As the temporary workaround, you may follow the KB to edit installation.yml, change memory_limit_percent from 85 to value like 60, then "Apply Change". The configuration can persist across Tanzu Platform Elastic Application Runtime upgrade. 

instance_groups:
- name: log_cache 
  jobs:
  - name: log-cache 
  properties:
    memory_limit_percent: 70

The product fix will be included in 10.3.2 release. If you have any questions, please reach out to Broadcom support.