Loggregator Performance downgrade in Tanzu Application Service (TAS) for VMs 2.7.x

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When using Tanzu Application Service (TAS) for VMs 2.7.12 - 2.7.15, loggregator components are showing high CPU usage causing log loss in the system.

You may see the following symptom (but not limited to) in your system when this issue occurs.

1. Intermittent 'cf push' or 'cf restart' failures.

2. Log and metrics loss in HealthWatch, metrics, cf tail, and other logging components.

3. Degraded performance (slowness) in the platform.

The cause of this issue is that in TAS for VMs 2.7.12 - 2.7.15, loggregator binaries are compiled with Golang 1.14.1, which has an upstream issue inside and may lead to deadlock in those binaries.

To identify which VM is being impacted, run the following command:

bosh -d <cf-deployment> instances --vitals

Then find out the instance with high CPU and SSH into it.

Once SSH'd, use "htop" to confirm which processes are using the most CPU.

Environment

Product Version: 2.7

Resolution

The fastest workaround is to restart whatever component that is using most CPU.

The long term fix is to upgrade to TAS for VMs 2.7.16, where all components are being compiled with Golang v1.14.2 or higher.