Gateway firewall Policy's show 'In Progress'

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• After upgrade to NSX-T 2.5.2.x, the gateway firewall section(s) show "In Progress"
• The in progress status, when clicked shows edge node(s) which don't have Tier-0 Gateways deployed on them and the following error:
  • Status is in-progress on X transport nodes.
  • Note: X refers to the number of transport nodes the message refers to.
• Also, if you go to Networking, Tier-0 Gateways, then click the status of the Tier-0 Gateway, this will also show a number of transport nodes In Progress, with their UUID's, see image:
• Log in as admin user on the NSX-T edge node(s), run the command get services, this shows the router service stopped, see image:
• Log in as root user on the NSX-T edge node(s), run the command: 
  • /opt/vmware/nsx-nestdb/bin/nestdb-cli --cmd get vmware.nsx.nestdb.ProcessedBarrierMsg
• Result:
  • Object type: vmware.nsx.nestdb.ProcessedBarrierMsg
    
    Object type: vmware.nsx.nestdb.ProcessedBarrierMsg
            {u'vertical_id': 'VERTICAL_ID_DHCP_SERVER_EDGE', u'barrier_num': 5173L}
  •        {u'vertical_id': 'VERTICAL_ID_DFW', u'barrier_num': 5173L}
  • Object type: vmware.nsx.nestdb.ProcessedBarrierMsg
  •        {u'vertical_id': 'VERTICAL_ID_L3_EDGE_AGENT', u'barrier_num': 5173L}
  • Object type: vmware.nsx.nestdb.ProcessedBarrierMsg
  •         {u'vertical_id': 'VERTICAL_ID_L3_EDGE_ROUTING', u'barrier_num': 5107L}.   
  • The result above, in bold we can see the issue here for the VERTICAL_ID_L3_EDGE_ROUTING
• And the next command lists the barrier message that each vertical should have:
  • /opt/vmware/nsx-nestdb/bin/nestdb-cli --cmd get vmware.nsx.nestdb.BarrierMsg
  • Object type: vmware.nsx.nestdb.BarrierMsg
  •         {u'desired_state_version': 5173L, u'id': {u'right': 0, u'left': 0}}
• As we see above the vertical VERTICAL_ID_L3_EDGE_ROUTING, does not have the correct barrier number.

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

The are two issues here:

• These edge nodes may have had a Tier-0 Gateway deployed on them at some stage and then removed. When this happens the barrier state does not get updated or removed and this causes the issue.
• Since 2.5.2.x the barrier does not get updated, even if there is or was never a Tier-0 Gateway deployed on the edge node.

Resolution

This issue is resolved in NSX-T 3.1.1.

Workaround:
Create a Tier-0 Service Router on the edge node(s) reporting the error. 
This will require an uplink or loopback interface to be created on all of the edge nodes.

If this does not work, please log a support request with GSS and refer to this KB.