Diego cells take 30 minutes to drain during TAS foundation upgrade

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service
• VMware NSX

Issue/Introduction

During Tanzu Application Service (TAS) foundation upgrades, Diego cells occasionally experience extended drain times, taking approximately 30 minutes to complete. This behavior exceeds the expected 10-minute threshold (Rep process drain timeout), significantly impacting the overall upgrade duration.

Environment

• Tanzu Application Service (TAS) for VMs
• NSX-T / NSX-T Tile
• Applicable to foundations utilizing NSX-T integration

Cause

The nsx-node-agent job initiates a drain operation on Diego cells with a hard-coded timeout of 1800 seconds (30 minutes). When the cleanup operation encounters delays—such as management plane sluggishness or Distributed Firewall (DFW) rule limitations on the transport nodes—the process hangs until it hits this limit.

Resolution

To resolve this delay, modify the NSX-T tile configuration to align with faster operational requirements:

1. Navigate to the NSX-T tile configuration in Ops Manager.
2. Locate the setting: Timeout time in seconds of removing all ovs ports.
3. Change the value to 600 (10 minutes).
4. Save and apply changes to the TAS foundation.

Troubleshooting: If the issue persists, check the ESXi transport nodes for the following alarm: distributed_firewall.dfw_rules_limit_per_host_exceeded.

Exceeding DFW rules per host causes the management plane to become unresponsive to cleanup commands during cell drainage. Monitor and optimize DFW rule limits to ensure the management plane synchronizes changes efficiently during upgrade events.

Additional Information

• For further assistance, see Contact Broadcom Support.