TKGI cluster network policy in validation fail status after creation

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When creating a Network Policy on a TKGI cluster where NSX is in policy API, you can see a NCP error like the following "ncp/error: NETWORK_POLICY_VALIDATION_FAILED."

From NCP logs you can see the following

"nsx_ujo.ncp.k8s.network_policy_controller Network policy ##### under namespace #### not created. Reason: Number of expressions per group exceeds NSX limit: Number of expressions exceed same member type NSX limit of 5 in target_group "

Cause

When enable_mixed_expression_groups is set to False, Network Policies that use multi-selectors criteria in Ingress/Egress rules are realized with dynamic NSX groups that are not affected by any performance degradation in calculating the actual members. However, the rules can be enforced on only up to 5 namespaces, depending on the other criteria defined in the Network Policy. If the Network Policy affects more than 5 namespaces at any point in time, it will be annotated with "ncp/error: NETWORK_POLICY_VALIDATION_FAILED"

 

The 5 criteria limit applies to both namespace and pod selectors. 

Resolution

enable_mixed_expression will be helpful for those cases where there are PodSelector and NamespaceSelector in the network policies, and the scale limitations have been resolved in NSX 4.2.0 (from NCP 4.2.0 release notes: https://docs.vmware.com/en/VMware-NSX-Container-Plugin/4.2.0/rn/vmware-nsx-container-plugin-420-release-notes/index.html

enable_mixed_expressions flag is an enhancement to bypass the 5-criteria limit when there are both pod and namespace selectors, but for the pod selector the limit is always 5.

The flag enable_mixed_expression_groups is enabled by default for Kubernetes Network Policies using multi-selectors in Policy mode.