BBR fails to restore C2C networking policies when NSX-T Plug-in Tile is present

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

Symptoms:
Container-to-Container (c2c) Networking policies aren't in effect.

Note: cf networking-policies will show the policies as existing and functional. However, it's misleading and the concrete policies are not reflected on the NSX-T Manager. For example, a policy may allow application A to communicate with application B, but NSX-T will block any traffic between the applications.

Environment

Cause

The VMware NSX-T Container Plug-in (NCP) Tile's daemon, is confused by the policy server's cache and assumes that the networking policy already exists. 

This leads to the following issues: 

• NCP queries the network policy server every 5 seconds to retrieve all available C2C Network policies
• The Network policy server caches data
• NCP 2.3 will modify NSX-T firewall rule(s) if the current query result is different from the previous one

As a result, in the event of restore, the NCP daemon no longer has an accurate state of the networking policies and doesn't recreate the missing networking policies.

NCP 2.4 and above use a different mechanism to avoid this failure.

Resolution

Restarting NCP (VMware NSX-T Container Plug-in Tile's dæmon) will force it to recreate the C2C rules.

To do so, one needs to obtain the BOSH credentials. Follow the Advanced Troubleshooting with the BOSH CLI to use the BOSH CLI.
 
Note: cf-### is the name of the PAS deployment which can be retrieved through bosh deployments.

Issue the Restart command to restart the NCP dæmon on the Diego Database instances: 

bosh -d cf-### restart diego_database

Note: On Small Footprint PAS installations, restart the control instance rather than diego_database  instances.

 

After the Diego Database instances have successfully restarted, the c2c Networking policies will be recreated in a few minutes.