NSX-T missing Firewall Sections and Rules

Products

• VMware vDefend Firewall

Issue/Introduction

• When you query DFW firewall, some sections and rules are not displayed. 
• NSX Manager Syslog will display the following error:

 <179>1 2019-08-23T12:17:54.560Z host NSX 23666 DISTRIBUTEDSERVICES [nsx@6876 comp="nsx-manager" errorCode="MP96321" subcomp="manager"] validateSectionExists: DS Section ########-####-####-####-########## not found
 <182>1 2019-08-23T12:17:54.560Z host NSX 23666 - [nsx@6876 audit="true" comp="nsx-manager" subcomp="manager"] UserName:'admin' ModuleName:'common-services' Operation:'GET@/api/v1/firewall/sections/########-####-####-####-##########'Operation status: 'failure' Error: The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive.

• Running an API GET call against the manager will return in error 600

root@host:~# curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/e61c7889-####-####-####-d08097dg67a3
{
  "httpStatus" : "NOT_FOUND",
  "error_code" : 600,
  "module_name" : "common-services",
  "error_message" : "The requested object : ########-####-####-####-##########a3 could not be found. Object identifiers are case sensitive."
}

Environment

VMware NSX-T Data Center

Cause

This is due to the inconsistency between the MP nodes and Corfu.

To identify the issue,  run the following steps:

• Get the Total number of firewall sections and rules across all MP nodes
• In the below logs, when we do a unique search for word count
  
  <179>1 2019-08-23T12:17:54.560Z host NSX 23666 DISTRIBUTEDSERVICES [nsx@6876 comp="nsx-manager" errorCode="MP96321" subcomp="manager"] validateSectionExists: DS Section ########-####-####-####-########## not found
<182>1 2019-08-23T12:17:54.560Z host NSX 23666 - [nsx@6876 audit="true" comp="nsx-manager" subcomp="manager"] UserName:'admin' ModuleName:'common-services' Operation:'GET@/api/v1/firewall/sections/########-####-####-####-##########'Operation status: 'failure' Error: The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive.
  
  Doing a unique count we have 284 Firewall Sections reporting these errors
  
  egrep "MP96321" nsx_manager_*/var/log/syslog | awk -F ' ' '{print $14}' | sort -n | uniq | wc -l
284
  
  
• When you query the nodes which are not hosting the manger VIP, these firewall details will show up  via cli
  
  curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/########-####-####-####-##########
{
  "locked" : false,
  "comments" : "Default section unlock comment",
  "lock_modified_by" : "admin",
  "lock_modified_time" : 1565776171230,
  "autoplumbed" : false,
  "enforced_on" : "VIF",
  "tcp_strict" : false,
  "resource_type" : "FirewallSection",
  "id" : "########-####-####-####-########## 3",
  "display_name" : "########-####-####-####-##########",
  "section_type" : "LAYER3",
  "stateful" : true,
  "rule_count" : 3,
  "is_default" : false,
  "_create_user" : "admin",
  "_create_time" : 1565776171229,
  "_last_modified_user" : "admin",
  "_last_modified_time" : 1565776262895,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_revision" : 3
  
  
• Firewall details will not show up for  the node hosting the VIP
  
  # curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/########-####-####-####-##########
{
  "httpStatus" : "NOT_FOUND",
  "error_code" : 600,
  "module_name" : "common-services",
  "error_message" : "The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive."
}
   

Resolution

Restart proton service on the manager hosting the VIP

restart service manager