Service Metrics in RabbitMQ tile leaves zombie (defunct) processes over time

Products

• VMware RabbitMQ

Issue/Introduction

Symptoms:
This article addresses an issue with the rabbitmq-server VMs spawning multiple child processes after upgrading the tile. 

The continual zombie process creation puts a strain on CPU and memory resources.

Environment

Cause

A zombie process appears when a child process exits and the parent does not handle the SIGCHLD signal properly. In such situation, the process is in a terminated state, but still has an entry in the process table. Normally, init can “reap” a zombie process it adopts. 

A new version of service-metrics (1.10.0) was introduced that uses BOSH Process Manager (BPM), which is used to isolate jobs. In this scenario, there is no init process, so the zombies are never reaped and build up over time.

This bug is not in Rabbit tile itself, but in the service-metrics job. The new version of service metrics is using BPM which causes the issue.

Resolution

The workaround is to run monit stop service-metrics and leave the service unmonitored.

$ bosh -d p-rabbitmq-f49918be0a08ffa98e5c ssh rabbitmq/INDEX
$ monit stop service-metrics

Another option is to periodically stop and start the service metrics service.

Note: On-Demand instances are also affected by this issue and the recommended workaround is the same.