Post upgrade of NSX-T, the NSX-T manager may loss connectivity
Article ID: 322538
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
- You just upgraded from NSX-T 3.2.1 or lower to a version higher than 3.2.1.
- If the NSX-T manager is then migrated, it may loss network connectivity.
- The NSX-T manager resides on a host prepared for NSX-T.
- Checking in the dataplane (on the ESXi host) we see that the NSX-T manager has a DFW slot 2 filter applied to it:
root@esx:~] summarize-dvfilter
...
world 4436057 vmm0:nsxmgr01 vcUuid:'50 3b 35 b1 8a 11 fc 2a-66 f4 9c 74 ea 8a 25 64'
port 67108977 nsxmgr01.eth0
vNic slot 2
name: nic-4436057-eth0-vmware-sfw.2 --->>> DFW slot 2 filter attached
agentName: vmware-sfw
state: IOChain Attached
vmState: Attached
failurePolicy: failClosed
serviceVMID: 3
filter source: Dynamic Filter Creation
moduleName: nsxt-vsip-20737187
...
Note: nsxmgr01 is the NSX-T manager.
...
world 4436057 vmm0:nsxmgr01 vcUuid:'50 3b 35 b1 8a 11 fc 2a-66 f4 9c 74 ea 8a 25 64'
port 67108977 nsxmgr01.eth0
vNic slot 2
name: nic-4436057-eth0-vmware-sfw.2 --->>> DFW slot 2 filter attached
agentName: vmware-sfw
state: IOChain Attached
vmState: Attached
failurePolicy: failClosed
serviceVMID: 3
filter source: Dynamic Filter Creation
moduleName: nsxt-vsip-20737187
...
- Reviewing the rules of the filter, we see the following WARNING:
/bin/vsipioctl getrules -f nic-4436057-eth0-vmware-sfw.2 -s
...
ruleset mainrs {
# PRE_FILTER rules
rule 7259 at 73, 972 evals, 972 hits, 972 sessions, in 19513 out 18445 pkts, in 4025968 out 19385689 bytes
rule 7218 at 79, 11 evals, 11 hits, 1 sessions, in 1500643 out 1504682 pkts, in 60025744 out 555353553 bytes
rule 7278 at 125, 540 evals, 540 hits, 536 sessions, in 585 out 585 pkts, in 44460 out 44460 bytes
rule 7891 at 126, 3561 evals, 3561 hits, 3561 sessions, in 3560 out 3561 pkts, in 470473 out 286329 bytes
rule 7892 at 128, 18942 evals, 18942 hits, 18942 sessions, in 213750 out 222909 pkts, in 117132007 out 31381466 bytes
rule 7879 at 168, 14 evals, 14 hits, 12 sessions, in 121 out 112 pkts, in 17543 out 165969 bytes
# FILTER (APP Category) rules
rule 3 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 3 at 2, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 4 at 3, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 2 at 4, 8290072 evals, 8275507 hits, 1 sessions, in 4988034 out 3290124 pkts, in 278454036 out 176915257 bytes
}
ruleset mainrs_L2 {
# FILTER rules
rule 1 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
}
}
2023-01-28T05:17:47.976Z WARN pool-94-thread-1 TransactionConsumer 1587 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] Unable to find VifStateHandler in ufo message cache: uuid {
c3d1f056-db8e-44a1-af4f-520dfe99f8f1
}
...
ruleset mainrs {
# PRE_FILTER rules
rule 7259 at 73, 972 evals, 972 hits, 972 sessions, in 19513 out 18445 pkts, in 4025968 out 19385689 bytes
rule 7218 at 79, 11 evals, 11 hits, 1 sessions, in 1500643 out 1504682 pkts, in 60025744 out 555353553 bytes
rule 7278 at 125, 540 evals, 540 hits, 536 sessions, in 585 out 585 pkts, in 44460 out 44460 bytes
rule 7891 at 126, 3561 evals, 3561 hits, 3561 sessions, in 3560 out 3561 pkts, in 470473 out 286329 bytes
rule 7892 at 128, 18942 evals, 18942 hits, 18942 sessions, in 213750 out 222909 pkts, in 117132007 out 31381466 bytes
rule 7879 at 168, 14 evals, 14 hits, 12 sessions, in 121 out 112 pkts, in 17543 out 165969 bytes
# FILTER (APP Category) rules
rule 3 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 3 at 2, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 4 at 3, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
rule 2 at 4, 8290072 evals, 8275507 hits, 1 sessions, in 4988034 out 3290124 pkts, in 278454036 out 176915257 bytes
}
ruleset mainrs_L2 {
# FILTER rules
rule 1 at 1, 0 evals, 0 hits, 0 sessions, in 0 out 0 pkts, in 0 out 0 bytes
}
}
2023-01-28T05:17:47.976Z WARN pool-94-thread-1 TransactionConsumer 1587 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] Unable to find VifStateHandler in ufo message cache: uuid {
c3d1f056-db8e-44a1-af4f-520dfe99f8f1
}
Environment
VMware NSX-T Data Center
Cause
There is an internal group used for system VM's, this group is used to add system VM's to the DFW exclusion list.
This issue occurs when the NSX-T manager is not added to this group.
This issue occurs when the NSX-T manager is not added to this group.
Resolution
This issue is resolved in VMware NSX 3.2.3 and 4.1.1, available at VMware downloads.
Workaround:
You can preform either of the below workarounds:
Workaround:
You can preform either of the below workarounds:
- If this is impacting a single NSX-T manager, the cluster is still up and the UI is accessible, you can add a new DFW rule which will allow communications to the impacted NSX-T manager.
Feedback
Yes
No
Powered by
