Upgrading NSX-T manager from 2.1.0.0 to 2.2.0.0 reports "General Error has occurred" on Firewall's General UI section.

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

It has been observed that after upgrading NSX manager (proton) from 2.1.0.0 (NSX-T DGo) to 2.2.0.0 (NSX-T Equinox), all firewall section (layer3 and layer 2) were not displayed on UI and in listSections API response.
Apparently, UI and API reports "General Error has occurred" on Firewall's General or Ethernet tabs and/or JSON response with a NullPointerException in NSX manager logs (nsxapi.log) in listSections API workflow.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

The NSX-T DFW allows the consumption of group references (like NSGroups, Logical Switches, IP Sets, ect.) as sources, destinations, and the appliedTo field.
When a firewall rule is created using these group references, DFW validates the existence of these entities. However, the DFW does not manage any future changes or updates made to the referenced object. Thus, if a user deletes a referenced entity after rule creation, then the firewall rule will contain a stale reference. 

During an upgrade of NSX, the DFW re-validates the existence of these group references. This attempt to revalidate the logical objects of the DFW results in the error: "General Error has occurred."

Resolution

There are couple ways customer can migrate firewall entities from 2.1 (DGo) to 2.2 (EQ) without experiencing above issue.
Please refer to workaround section.

Workaround:
There are couple ways customer can migrate firewall entities from 2.1 (DGo) to 2.2 (EQ) without experiencing above issue:

In NSX-T  2.1:
User can invoke listSections API (sample below) before upgrading NSX Manager and look for firewall sections consuming invalid references ("is_valid" : false) in JSON API response. Then, perform one of the following operations best suited to keep his/her security posture unscathed:

• Delete Firewall Section (along with rules) which has all references in appliedTos as invalid.
• Or Update Firewall Section to sanitize appliedTo list to contain only valid references (Note: Emptying appliedTo essentially mean applied to ALL)

API Sample:
GET  /api/v1/firewall/sections?type=[LAYER3|LAYER2]

{

"cursor": "003616a93c39-5822-469e-b0be-70728153642eFirewallSection",
"sort_by": "position",
"result_count": 2,
"results": [
{
"resource_type": "FirewallSection",
"description": "protects app VMs",
"id": "65a60890-6f48-418d-a7d5-5aaf65fc4e5b",
"display_name": "app-firewall",
"section_type": "LAYER3",
"applied_tos": [
{
"target_display_name": "app-switch-port",
"is_valid": false,
"target_type": "LogicalPort",
"target_id": "1d9fb5cb-0344-4d7f-899a-afd93276899f"
}
],
"is_default": false,
"stateful": true,
"rule_count": 0,
"locked": true,
"comments": "Locked the section",
"lock_modified_by": "admin",
"lock_modified_time": "14458940331000",
"_last_modified_user": "admin",
"_system_owned": false,
"_last_modified_time": 1445894033107,
"_create_user": "admin",
"_create_time": 1445894033107,
"_revision": 0
}
]
}

 

In NSX-T 2.2:

If NSX is already upgraded to Equinox (2.2) and you have encountered this issue, then a jar file patch can be applied on NSX-T manager. Once the patch is applied, reboot the manager, and DFW objects and UI should come up properly. 
Please open a case with VMware support to acquire the jar file patch.