Default isolation rules created by NCP accidentally block legitimate traffic on TAS on NCP version 4.1

Products

• VMware Tanzu Application Service

Issue/Introduction

• Post upgrade of NSX-NCP tile in TAS environment falling under range of 4.1.0 - 4.1.2, observing legitimate traffic between applications and cloud controller VMs are getting blocked by default isolation deny rules.
• NSX-NCP is configured in "Policy" mode.
• Before upgrade of NSX-NCP the aforementioned traffic was getting allowed. 

Environment

VMware NSX Container Plugin, Tanzu Application Service

Cause

• Before NCP 4.1.0 the default isolation DFW rules created by NCP in TAS environment contained 1 outbound allow rule for Cloud Foundry VMs (Dopplers, TCP routers etc.) and another default deny rule. These 2 rules were applied to a dynamic security group containing all the logical-switch-ports of the containers.  

• Starting from NSX-NCP 4.1.0 onwards these default isolation rules are changed in the following manner,
• a. 1 outbound allow rule for Cloud Foundry VMs, 'allow-selected-cf-vms-static'
• b. 1 deny inbound rule, 'deny-all-ingress'
• c. 1 deny outbound rule, 'deny-all-egress'

• All these rules are applied on DFW, means these rules will be present on all the NSX-T connected VMs and containers, including CF Controller VMs. Also the "direction" field on these deny rules are set as IN_OUT inspecting traffic on both the direction.
• Due to this settings some legitimate traffic between applications and cloud controller VMs gets dropped by these deny rules.

Resolution

This issue is resolved on NCP 4.1.2.2 onwards. This is also documented under Resolved issues of the said release.

Workaround:

Update the two deny firewall rules in the default isolation section for the foundation.
For the rule with source equal to the container CIDR and destination ANY (deny-all-egress), the rule's direction must be changed from IN_OUT to OUT.
For the rule with destination equal to the container CIDR and source any (deny-all-ingress), the rule's direction must be changed from IN_OUT to IN.
If the TAS foundation is configured to use a NSX principal identity, this operation must be performed via API specifying the 'X-Allow-Overwrite:True' header.