NSX-T missing Firewall Sections and Rules

Products

• VMware NSX

Issue/Introduction

Symptoms:

• When you query DFW firewall, some sections and rules are not displayed. 
• NSX Manager Syslog will display the following error:

 <179>1 2019-08-23T12:17:54.560Z host NSX 23666 DISTRIBUTEDSERVICES [nsx@6876 comp="nsx-manager" errorCode="MP96321" subcomp="manager"] validateSectionExists: DS Section ########-####-####-####-########## not found
 <182>1 2019-08-23T12:17:54.560Z host NSX 23666 - [nsx@6876 audit="true" comp="nsx-manager" subcomp="manager"] UserName:'admin' ModuleName:'common-services' Operation:'GET@/api/v1/firewall/sections/########-####-####-####-##########'Operation status: 'failure' Error: The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive.

• Running an API GET call against the manager will return in error 600

root@host:~# curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/e61c7889-28e5-4d97-bfae-d08097dg67a3
{
  "httpStatus" : "NOT_FOUND",
  "error_code" : 600,
  "module_name" : "common-services",
  "error_message" : "The requested object : ########-####-####-####-##########a3 could not be found. Object identifiers are case sensitive."
}

Environment

VMware NSX-T Data Center

Cause

This is due to the inconsistency between the MP nodes and Corfu.

To identify the issue,  run the following steps:

• Get the Total number of firewall sections and rules across all MP nodes
• In the below logs, when we do a unique search for word count
•  <179>1 2019-08-23T12:17:54.560Z host NSX 23666 DISTRIBUTEDSERVICES [nsx@6876 comp="nsx-manager" errorCode="MP96321" subcomp="manager"] validateSectionExists: DS Section ########-####-####-####-########## not found
     <182>1 2019-08-23T12:17:54.560Z host NSX 23666 - [nsx@6876 audit="true" comp="nsx-manager" subcomp="manager"] UserName:'admin' ModuleName:'common-services' Operation:'GET@/api/v1/firewall/sections/########-####-####-####-##########'Operation status: 'failure' Error: The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive.
  
  Doing a unique count we have 284 Firewall Sections reporting these errors
  
     egrep "MP96321" nsx_manager_*/var/log/syslog | awk -F ' ' '{print $14}' | sort -n | uniq | wc -l
     284
• When you query the nodes which are not hosting the manger VIP, these firewall details will show up  via cli
• ~# curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/########-####-####-####-##########
  {
    "locked" : false,
    "comments" : "Default section unlock comment",
    "lock_modified_by" : "admin",
    "lock_modified_time" : 1565776171230,
    "autoplumbed" : false,
    "enforced_on" : "VIF",
    "tcp_strict" : false,
    "resource_type" : "FirewallSection",
    "id" : "########-####-####-####-########## 3",
    "display_name" : "########-####-####-####-##########",
    "section_type" : "LAYER3",
    "stateful" : true,
    "rule_count" : 3,
    "is_default" : false,
    "_create_user" : "admin",
    "_create_time" : 1565776171229,
    "_last_modified_user" : "admin",
    "_last_modified_time" : 1565776262895,
    "_system_owned" : false,
    "_protection" : "NOT_PROTECTED",
    "_revision" : 3
• Firewall details will not show up for  the node hosting the VIP
• # curl -k -H 'X-NSX-Username:admin' -H 'X-Nsx-Groups: superusers' -X GET http://localhost:7440/nsxapi/api/v1/firewall/sections/########-####-####-####-##########
  {
    "httpStatus" : "NOT_FOUND",
    "error_code" : 600,
    "module_name" : "common-services",
    "error_message" : "The requested object : ########-####-####-####-########## could not be found. Object identifiers are case sensitive."
  }
   

Resolution

This issue will be fixed in the future NSX-T release 

Workaround:
Restart proton service on the manager hosting the VIP
nsxmgR> restart service manager