VMware NSX DFW Default isolation rules created by NCP may block legitimate traffic on TAS

Products

• VMware NSX
• VMware Tanzu Application Service

Issue/Introduction

• You are running TAS with NCP in policy mode.
• You recently upgraded NCP to versions 4.1.0, 4.1.1 and 4.1.2.
  During the TAS upgrade, this may fail with error like in the CF log app-usage-server:
  [HEALTH/1] ERR Timed out after 3m0s (90 attempts) waiting for startup check to succeed: failed to make HTTP request to '/heartbeat/server_status' on port 8080: connection refused
  [CELL/1] ERR Failed after 3m0.27s: startup health check never passed.
  [CELL/SSHD/1] OUT Exit status 0
• In the VMware NSX-T DFW there are 2 default rules named  deny_all_ingress and deny_all_egress. The direction for these rules is IN_OUT.

Environment

• NCP 4.1.X
• TAS (any supported version)
• NSX (any supported version)

Cause

• In these NCP versions 4.1.0, 4.1.1 and 4.1.2, policy mode, the 2 default DFW rules deny_all_ingress and deny_all_egress, have a direction of IN_OUT, meaning they are applied in both directions.
• This can lead to communications errors between containers and other appliances deployed on the same NSX instance, such as TAS VMs.

Resolution

Resolution:
The issue is resolved in NCP versions 4.1.0.5, 4.1.1.5, 4.1.2.2, and 4.2.0

Workaround:
If you have encountered this issue after upgrade and traffic from TAS applications is being impacted, you can apply the following workaround:

• Update the two firewall rules in the default isolation section for the foundation.
• For the deny_all_egress rule with source equal to the container CIDR and destination ANY, the rule's direction must be changed from IN_OUT to OUT.
• For the deny_all_ingress rule rule with destination equal to the container CIDR and source any, the rule's direction must be changed from IN_OUT to IN.
• If the TAS foundation is configured to use a NSX principal identity, this operation must be performed via API specifying the X-Allow-Overwrite:True header.
• NCP restart is not required. In case of NCP restarts the rule direction won’t be updated back to IN_OUT.