Dynamic Application Security Groups not being enforced

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

A rare bug has been discovered in relation to Dynamic Application Security Groups (ASG) enforcement in Tanzu Application Service (TAS).
This bug is encouraged by the following:

• Frequent ASG creation/update/deletions/bindings/unbindings such as large scale rule configurations via the cf-mgmt utility
• A lower sync interval for policy-server as to speed up ASG change propagations

This is related to dynamic ASGs having a race condition between policy-server-asg-syncer and Cloud Controller API (CAPI). At a high level the policy-server last_updated timestamp previously had per second level precision, this created a race condition when multiple c2c policies were updated at nearly the same time, but the vxlan-policy-agent would only pick up the first update. The github issue can be found here.

Before attributing an issue to this potential bug, it is advisable to conduct standard troubleshooting procedures, please see this document for helpful instructions on how to get started.

Environment

Product Version: 2.13

Resolution

This bug has been patched by changing the policy-server last_updated timestamp to have microsecond precision starting in cf-networking version 3.35.0 +. This patch is available starting in the following TAS versions:

•  TAS v2.13.31+
•  TAS v4.0.12+
•  TAS v5.0.2+

​​
Until the patch can be consumed via a TAS upgrade, the following is interim solution.

Simply make an arbitrary ASG change to resolve the issue in any affected foundation, For example create/update an  ASG binding/definition.

cf create-security-group myasg-test myasg.json
cf bind-security-group myasg-test test-org --space test-space