BBR backup of TAS fails during unlock of bbr-usage-servicedb with "external networker up: exit status 1"

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Symptoms

• VMware Tanzu Application Service (TAS) BBR backups fail intermittently (approx. 80% of the time) during the post-backup-unlock phase.
• The error log reports: external networker encountered an error running 'up' action: exit status 1.
• Following the failure, the system/system/app-usage-server and system/system/app-usage-worker instances are often left in a stopped state.
• The app-usage-scheduler may crash multiple times before eventually succeeding or timing out.

 

Environment

• VMware Tanzu Application Service (TAS)
• VMware NSX Container Plugin (NCP) versions prior to 4.2.4
• Foundations with a high number of Diego Cells (typically >250)

Cause

The issue is caused by a performance bottleneck in the NSX Container Plugin (NCP) processing queue during the re-sync phase that follows a BBR backup.

1. Queue Congestion: After a BBR run, NCP must perform a full sync of all Actual LRPs (Local Runtime Processes). In large foundations, this queue can exceed 10,000 entries.
2. Redundant API Calls: In affected versions, NCP bypasses its cache and performs a search query to NSX for every segment port, adding approximately 500ms of latency per entry.
3. Timeout: The app-usage-scheduler attempts to start while the NCP queue is still clogged. Because NCP cannot provide network info in time, the container startup fails with a CNI timeout (exit status 1).
4. Error: Nsx-node-agent error with NCP01004 - Unable to retrieve network info for container, network interface for it will not be configured

Resolution

To resolve this issue, perform the following steps:

1. Upgrade NCP (Permanent Fix) Upgrade the NSX Container Plugin to version 4.2.4 or later. This version includes optimizations to the segment port cache expiration logic and skips redundant realization checks for unchanged ports, significantly reducing the time required to clear the sync queue.

2. Increase Unlock Timeout (Workaround) If an immediate upgrade is not possible, increase the allowed wait time for the usage service database to unlock:

1. Navigate to the BOSH job directory: /var/vcap/jobs/bbr-usage-servicedb.
2. Modify the unlock configuration to increase the timeout from the default 120 seconds to 1200 seconds.
3. This provides NCP sufficient time to process the clogged queue and provide the necessary network headers to the app-usage-scheduler.

3. Optimize NSX Manager Sizing Ensure that the NSX Manager VMs are sized appropriately (e.g., Extra-Large) to handle the high volume of API requests generated during a foundation-wide re-sync.