NSX Container Plugin reports "Operation not permitted. Incompatible anchor section FirewallSection" error when creating cluster with network-profile that uses Distributed Firewall section markers in Tanzu Kubernetes Grid Integrated Edition

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Your environment encounters the following scenario: 

• You are running a Tanzu Kubernetes Grid Integrated Edition (TKGI) foundation that runs with NSX-T networking.
• You have Distributed Firewalls (DFW) configured with section markers.
• You are using a network-profile for your TKGI clusters. For more information, refer to: Define DFW Section Markers.

For example:

{"cni_configurations":{"parameters":{"bottom_firewall_section_marker":"","enable_err_crd":true,"log_settings":{"log_dropped_traffic":true,"log_level":"INFO"},"top_firewall_section_marker":"","x_forwarded_for":"replace"},"type":"nsxt"},"lb_size":"small","master_vms_nsgroup_id":""}

You recently upgraded NSX-T. Creating new clusters fails when using that network-profile. 
 

ERROR

The bosh task <tasknum> --debug for the failed cluster task shows an issue with 'apply-addons' errand and coredns (kube-dns) deployment. It looks similar to this:

"result_output" = '{"instance":{"group":"apply-addons","id":""},"errand_name":"apply-addons","exit_code":1,"stdout":"Deploying /var/vcap/jobs/apply-specs/specs/coredns.yml\nserviceaccount/coredns created\nclusterrole.rbac.authorization.k8s.io/system:coredns created\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns created\nconfigmap/coredns created\ndeployment.apps/coredns created\nservice/kube-dns created\nWaiting for deployment \"coredns\" rollout to finish: 0 of 3 updated replicas are available...\nfailed to start all system specs after 1200 with exit code 1\n","stderr":"error: deployment \"coredns\" exceeded its progress deadline\n","logs":{"blobstore_id":"","sha1":

 

ERROR

When you check ncp.stdout.log from the master node in NSX Container Plugin (NCP), you see the following:

[nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" security="True" errorCode="NCP00034"] nsx_ujo.ncp.nsx.manager.nsxapi create_firewall_section failed, cause: Unexpected error from backend manager (['']) for POST api/v1/firewall/sections?operation=insert_after&id=: Operation not permitted. Incompatible anchor section FirewallSection/ for DFW firewall., args: ('hc-pks--section',), kwargs: {'description': 'Health Check Section', 'rules': None, 'tags': [{'scope': 'ncp/v[truncated]...

Environment

Product Version: 1.10

Resolution

This can happen when creating your Distributed Firewalls (DFW) rules with the use of the NSX-T Policy API and not through the Manager API.

The issue will likely occur when the DFW rules (created with Policy API) are also using section anchors as well.

Note: The NSX-T Policy API is not supported with TKGI as of version NSX-T 3.0.3.0.

To resolve this issue, recreate the DFW section using the Manager API (refer to Screenshot):



To prevent this from happening in the future, make sure your DFWs are created through the Manager API before upgrading NSX-T