How to Fix Queued BOSH Tasks Caused by bosh_exporter or other clients

Products

• Operations Manager

Issue/Introduction

By default, Tanzu Operations Manager BOSH Director is configured with 5 workers. Each worker can handle one task at a time (e.g., is, deployments, locks).

When external consumers—specifically monitoring tools like bosh_exporter - frequently poll the Director (e.g., GET /deployments), they can saturate all available workers. This results in critical operational tasks (like bosh deploy) remaining in a "queued" state for extended periods, delaying infrastructure management and emergency fixes.

 

Cause

The BOSH Director uses a task queue system. If the rate of incoming requests exceeds the processing capacity of the configured worker pool, the queue grows.

Resolution

Increase Worker Count

To handle higher concurrency, you must explicitly increase the number of workers in "Operations Manager > BOSH Director > Director Config > Director Workers". Recommended Value: from 5 to 10 or 15, depending on your platform need. 

Vertical Scaling (CPU/Memory)

As each additional worker is a process that consumes system resources, please scale up CPU/memory resource of BOSH Director VM so that the usage could remain at healthy level. 

Request Optimization

Before scaling horizontally worker count and resource, please review the demand side if the requests to BOSH Director can be reduced.