Route Registrar Memory Leak Mitigation

Products

• VMware Tanzu Application Service
• VMware Tanzu Application Service

Issue/Introduction

A memory leak was introduced to the route registrar job in routing release 0.305.0. The route registrar job is present on the following VMs: cloud_controller, diego_database, log_cache, loggregator_trafficcontroller, mysql_proxy, nfs_server, and uaa.

In a test environment, route registrar’s memory grew at the rate of 125 MB a day (3.7 GB a month).

Customers can monitor their VMs’ memory usage and restart the process as necessary. Routing release 0.315.0 has a fix and will be out in TAS soon.

Environment

Affected Versions

• Routing Release 0.305.0 - 0.314.0
• TAS 4.0.28 - 4.0.29
• TAS 5.0.18 - 5.0.19
• TAS 6.0.8 - 6.0.9
• TAS 10.0.0

Cause

 

Resolution

Running monit restart on the route_registrar job will free the extra memory. This will not impact uptime or disrupt service. Since the memory leak is slow growing, restarting the job will effectively mitigate the issue. Please see the following script as inspiration for the mitigation.

#!/bin/bash

vms=(
  "uaa"
  "cloud_controller"
  "log_cache"
  "nfs_server"
  "diego_database"
  "mysql_proxy"
  "loggregator_trafficcontroller"
)

for vm in "${vms[@]}"; do
  echo "Connecting to ${vm}..."
  if bosh ssh "${vm}" --command "sudo monit restart route_registrar"; then
    echo "Successfully restarted route_registrar on ${vm}."
  else
    echo "Failed to restart route_registrar on ${vm}."
  fi
done