Post upgrade of VMware NSX, the NSX manager may lose connectivity
search cancel

Post upgrade of VMware NSX, the NSX manager may lose connectivity

book

Article ID: 322538

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

 

  • You just upgraded from NSX-T 3.2.1 or lower to a version higher than 3.2.1.
  • If the NSX-T manager is then migrated, it may loss network connectivity.
  • The NSX-T manager resides on a host prepared for NSX-T.
  • Checking in the dataplane (on the ESXi host) we see that the NSX-T manager has a DFW slot 2 filter applied to it:
root@esx:~] summarize-dvfilter
...
world 4436057 vmm0:nsxmgr01 vcUuid:'50 3b 35 b1 ## ## ## ##-## ## ## ## ea 8a 25 64'
 port 67108977 nsxmgr01.eth0
  vNic slot 2
   name: nic-4436057-eth0-vmware-sfw.2 --->>> DFW slot 2 filter attached
   agentName: vmware-sfw
   state: IOChain Attached
   vmState: Attached
   failurePolicy: failClosed
   serviceVMID: 3
   filter source: Dynamic Filter Creation
   moduleName: nsxt-vsip-20737187
...

Note: nsxmgr01 is the NSX-T manager.

  • Reviewing the rules of the filter, we see the following WARNING:


/bin/vsipioctl getrules -f nic-4436057-eth0-vmware-sfw.2 -s 
...
ruleset mainrs {
  # PRE_FILTER rules
rule 7259 at 73, 972 evals, 972 hits, 972 sessions, in 19513 out 18445 pkts, in 4025968 out 19385689 bytes
rule 7218 at 79, 11 evals, 11 hits, 1 sessions, in 1500643 out 1504682 pkts, in 60025744 out 555353553 bytes
rule 7278 at 125, 540 evals, 540 hits, 536 sessions, in 585 out 585 pkts, in 44460 out 44460 bytes
rule 7891 at 126, 3561 evals, 3561 hits, 3561 sessions, in 3560 out 3561 pkts, in 470473 out 286329 bytes
rule 7892 at 128, 18942 evals, 18942 hits, 18942 sessions, in 213750 out 222909 pkts, in 117132007 out 31381466 bytes
rule 7879 at 168, 14 evals, 14 hits, 12 sessions, in 121 out 112 pkts, in 17543 out 165969 bytes

  # FILTER (APP Category) rules
rule 3 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 3 at 2, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 4 at 3, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 2 at 4, 8290072 evals, 8275507 hits, 1 sessions, in 4988034 out 3290124 pkts, in 278454036 out 176915257 bytes
}

ruleset mainrs_L2 {
  # FILTER rules
rule 1 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
}
}
2023-01-28T05:17:47.976Z WARN pool-94-thread-1 TransactionConsumer 1587 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] Unable to find VifStateHandler in ufo message cache: uuid {
  ########-####-####-####-########f8f1
}

Environment

VMware NSX 4.x
VMware NSX-T Data Center 3.x

Cause

There is an internal group used for system VM's, this group is used to add system VM's to the DFW exclusion list.
This issue occurs when the NSX-T manager is not added to this group.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3 and VMware NSX 4.1.1, available at Broadcom Downloads.

Workaround:

You can preform either of the below workarounds:
  1. If this is impacting a single NSX-T manager, the cluster is still up and the UI is accessible, you can add a new DFW rule which will allow communications to the impacted NSX-T manager. 
If you believe you have encountered this issue and are unable to implement the workaround(s) above, please open a support request with Broadcom Support and reference this KB.