Post upgrade of VMware NSX, the NSX manager may lose connectivity

Products

• VMware NSX

Issue/Introduction

 

• You just upgraded from NSX-T 3.2.1 or lower to a version higher than 3.2.1.
• If the NSX-T manager is then migrated, it may loss network connectivity.
• The NSX-T manager resides on a host prepared for NSX-T.
• Checking in the Dataplane (on the ESXi host) we see that the NSX-T manager has a DFW slot 2 filter applied to it:
  

  root@esx:~] summarize-dvfilter
  world ####057 vmm0:nsx_manager_name vcUuid:'50 3b 35 b1 ## ## ## ##-## ## ## ## ea 8a 25 64'
  port ####8977 nsx_manager_name.eth0
  vNic slot 2
  
  name: nic-####057-eth0-vmware-sfw.2 --->>> DFW slot 2 filter attached
  agentName: vmware-sfw
  state: IOChain Attached
  vmState: Attached
  failurePolicy: failClosed
  serviceVMID: 3
  filter source: Dynamic Filter Creation
  moduleName: nsxt-vsip-####7187
  
  Note: nsxmgr01 is the NSX-T manager.

• Reviewing the rules of the filter, we see the following WARNING:
  

  /bin/vsipioctl getrules -f nic-4436057-eth0-vmware-sfw.2 -s 
  ...
  ruleset mainrs {
    # PRE_FILTER rules
  rule ##59 at 73, 972 evals, 972 hits, 972 sessions, in 19513 out 18445 pkts, in 4025968 out 19385689 bytes
  rule ##18 at 79, 11 evals, 11 hits, 1 sessions, in 1500643 out 1504682 pkts, in 60025744 out 555353553 bytes
  rule ##78 at 125, 540 evals, 540 hits, 536 sessions, in 585 out 585 pkts, in 44460 out 44460 bytes
  rule ##91 at 126, 3561 evals, 3561 hits, 3561 sessions, in 3560 out 3561 pkts, in 470473 out 286329 bytes
  rule ##92 at 128, 18942 evals, 18942 hits, 18942 sessions, in 213750 out 222909 pkts, in 117132007 out 31381466 bytes
  rule ##79 at 168, 14 evals, 14 hits, 12 sessions, in 121 out 112 pkts, in 17543 out 165969 bytes
  
    # FILTER (APP Category) rules
  rule ###3 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
  rule ###3 at 2, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
  rule ###4 at 3, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
  rule ###2 at 4, 8290072 evals, 8275507 hits, 1 sessions, in 4988034 out 3290124 pkts, in 278454036 out 176915257 bytes
  }
  
  ruleset mainrs_L2 {
    # FILTER rules
  rule ###1 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
  }
  }
  2023-01-28T05:17:47.976Z WARN pool-94-thread-1 TransactionConsumer 1587 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] Unable to find VifStateHandler in ufo message cache: uuid {
    ########-####-####-####-########f8f1
  }

 

Environment

VMware NSX-T Data Center

VMware NSX

Cause

There is an internal group used for system VM's, this group is used to add system VM's to the DFW exclusion list.
This issue occurs when the NSX-T manager is not added to this group.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3

This issue is resolved in VMware NSX 4.1.1, available at Broadcom Downloads.

 

Workaround:

• If this is impacting a single NSX-T manager, the cluster is still up and the UI is accessible, you can add a new DFW rule which will allow communications to the impacted NSX-T manager. 
• • Add DFW Policy/Rules.

 

Note: If you believe you have encountered this issue and are unable to implement the workaround(s) above, please open a support request with Broadcom Support and reference this KB.