vxlan-policy-agent reports CHAIN_USER_DEL error (Device or resource busy) during poll-cycle

Products

• VMware Tanzu Platform Core
• VMware Tanzu Application Service
• Vmware Tanzu Platform - SM
• VMware Tanzu Platform - SaaS Components

Issue/Introduction

The cfnetworking.vxlan-policy-agent logs on a Diego Cell or Isolation Segment reporterrors similar to the following during its regular poll-cycle:

cfnetworking.vxlan-policy-agent.poll-cycle: 1 error occurred: 
* clean-up-asg-chains-matching: deleting chain asg-xxxxxxxxxxxxxxxxxxxx from table filter: 
delete old chain: iptables call: running [/sbin/iptables -t filter -X asg-xxxxxxxxxxxxxxxxxxxx --wait]: 
exit status 4: iptables v1.8.7 (nf_tables): CHAIN_USER_DEL failed (Device or resource busy)

Environment

• VMware Tanzu Application Service (TAS)
• VMware Tanzu Isolation Segment
• Tanzu Platform Core

Cause

This error is typically caused by a race condition in the cleanup of Application Security Group (ASG) chains.

When an application container is destroyed or moved, the vxlan-policy-agent attempts to delete the corresponding iptables chain. If the iptables binary or the nf_tables kernel subsystem is currently processing a rule update or holding a lock on the filter table at the exact moment the delete command is issued, the system returns "Device or resource busy."

Resolution

In the vast majority of scenarios, these errors are benign and do not require manual intervention.

1. Automatic Retry: The vxlan-policy-agent operates on a continuous polling cycle. If a chain deletion fails due to a "busy" resource, the agent will attempt the cleanup again in the next cycle (typically every 30 seconds).
2. Impact Assessment: Because the ASG chain is tied directly to a specific container instance that no longer exists, the orphaned chain does not affect traffic flow for active containers. Traffic cannot be routed to a non-existent container interface, meaning there is no security or performance impact during the period the chain remains in the table.
3. Verification:
   • Monitor the logs to see if the specific chain ID (e.g., asg-8612a12...) disappears in subsequent log entries.
   • If the error persists for the same chain ID for more than 15 minutes, it may indicate a stale process lock. In such cases, a restart of the agent is recommended

monit restart vxlan-policy-agent

Additional Information

Related article: 

Apply changes fails in Isolation segment tile due to vxlan-policy-agent error

R&D is updating the error logging so that the benign errors are classified as "WARNING" level. Subscribe to this article for updates.