Concourse worker nodes enter stalled state with "max containers reached" error

Products

• VMware Tanzu Platform Core
• Concourse for VMware Tanzu

Issue/Introduction

Worker nodes in a Concourse environment transition to a stalled state, often within 24 hours of uptime. This typically occurs when container limits are reached, preventing the reporting of active container status to the Web/ATC nodes.

 

• Workers show a stalled state when running fly workers.
• Web/ATC logs contain repeated errors: "error":"run check: find or create container on worker [Worker-ID]: max containers reached".
• Worker logs (Guardian/Containerd) show: "message":"guardian.api.garden-server.create.failed","data":{"error":"max containers reached"}.
• BOSH DNS logs on the worker node show upcheck failures: WARN - upcheck failed (on tcp: dial tcp ###.###.###.###:53: i/o timeout).
• Communication timeouts occur between the worker and TSA/ATC (e.g., use of closed network connection).
• Watch for memory spikes on the worker nodes.

Environment

• Product: Concourse for VMware Tanzu
• Versions: 7.x, 8.x
• Container Runtime: Guardian or Containerd

Cause

A race condition or imbalance exists in container placement. If the placement strategy does not prioritize the max-containers limit, the ATC continues scheduling "Check" containers on workers already at capacity. The resulting high-load condition causes the worker to become unresponsive and stop sending heartbeats and is eventually marked as stalled.

Resolution

Modify the container placement strategies in the Concourse manifest to ensure the ATC respects maximum container limits before evaluating volume locality.

1. Open the Concourse deployment manifest.
2. Locate the check_container_placement_strategies and container_placement_strategy properties.
3. Update the order to place limit-max-containers at the beginning of the list:

# Strategy for Resource Check containers
check_container_placement_strategies:
  - limit-max-containers
  - limit-active-containers
  - fewest-build-containers

# Strategy for Task/Step containers
container_placement_strategy: limit-max-containers,volume-locality,fewest-build-containers

4. Deploy changes using BOSH: bosh -d concourse deploy manifest.yml
5. Prune existing stalled workers to clear the cluster state: fly -t <target> prune-worker -a

Workaround

If immediate manifest changes are not possible:

• Increase Worker Memory: Provide more overhead for the worker to remain responsive during high container density.
• Decrease Max Containers: Lower the --worker-max-containers flag to a value that matches the VM's actual resource capacity to prevent "hard" stalls.

Additional Information

• To receive updates on this issue, subscribe to this article (Reference: KB 275360).
• For further assistance, contact Support. See Contact Broadcom Support. To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.