PSOD due to Excessive L7 Attributes in VMotion Import blob
search cancel

PSOD due to Excessive L7 Attributes in VMotion Import blob

book

Article ID: 377381

calendar_today

Updated On:

Products

VMware vDefend Firewall

Issue/Introduction

PSOD can occur on an ESXi host if a vmotion import blob contains an extreme number (millions) of L7 attributes.

The following backtrace is observed in the LogEFI.log:

2024-08-02T01:50:03.050Z cpu29:2097635)ESC[45mESC[33;1mVMware ESXi 7.0.3 [Releasebuild-23794027 x86_64]ESC[0m
LINT1/NMI (motherboard nonmaskable interrupt), undiagnosed. This may be a hardware problem; please contact your hardware vendor.
2024-08-02T01:50:03.103Z cpu29:2097635)cr0=0x8001003d cr2=0xffffc0009877f000 cr3=0x9bbee000 cr4=0x10216c
2024-08-02T01:50:03.110Z cpu29:2097635)FMS=06/55/7 uCode=0x5003605
*PCPU29:2097635/PSEventHelper
PCPU 0: VSVSVVUVVSUSVVVUVVVVVVVVVVVVVSVVVSVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
PCPU 64: VVVVVVVVVVVVVVVV
2024-08-02T01:50:03.160Z cpu29:2097635)Code start: 0x420033000000 VMK uptime: 44:07:49:55.015
2024-08-02T01:50:03.184Z cpu29:2097635)0x4529001d2d10:[0x4200330ff107]PanicvPanicInt@vmkernel#nover+0x327 stack: 0x4529001d2de8
2024-08-02T01:50:03.207Z cpu29:2097635)0x4529001d2de0:[0x4200330ff660]Panic_NoSave@vmkernel#nover+0x4d stack: 0x4529001d2e40
2024-08-02T01:50:03.228Z cpu29:2097635)0x4529001d2e40:[0x4200330fbdba]NMI_Interrupt@vmkernel#nover+0x50f stack: 0x0
2024-08-02T01:50:03.250Z cpu29:2097635)0x4529001d2f00:[0x420033153392]IDTNMIWork@vmkernel#nover+0x7f stack: 0x420047400000
2024-08-02T01:50:03.269Z cpu29:2097635)0x4529001d2f20:[0x4200331546dc]Int2_NMI@vmkernel#nover+0x19 stack: 0x0
2024-08-02T01:50:03.289Z cpu29:2097635)0x4529001d2f40:[0x42003314e067]gate_entry@vmkernel#nover+0x68 stack: 0x0
2024-08-02T01:50:03.312Z cpu29:2097635)0x45394f19be00:[0x42003310283a]RefCountBlock@vmkernel#nover+0x66 stack: 0x43063aa42e7e
2024-08-02T01:50:03.335Z cpu29:2097635)0x45394f19be10:[0x4200332ac24d]Portset_LockExclWithRA@vmkernel#nover+0x42 stack: 0x2
2024-08-02T01:50:03.359Z cpu29:2097635)0x45394f19be40:[0x4200332ac7f6]vmk_PortsetAcquireByName@vmkernel#nover+0x41f stack: 0x0
2024-08-02T01:50:03.383Z cpu29:2097635)0x45394f19bec0:[0x4200349df01e]hbVlanEventCB@(heartbeat)#<None>+0x1f stack: 0x43063aaf7390
2024-08-02T01:50:03.408Z cpu29:2097635)0x45394f19bee0:[0x4200332afd23]PortsetFireEventCB@vmkernel#nover+0x170 stack: 0x4200332afd12
2024-08-02T01:50:03.432Z cpu29:2097635)0x45394f19bf40:[0x4200330da2f9]HelperQueueFunc@vmkernel#nover+0x1d2 stack: 0x45394f1a0b48
2024-08-02T01:50:03.454Z cpu29:2097635)0x45394f19bfe0:[0x4200333b4d55]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0
2024-08-02T01:50:03.476Z cpu29:2097635)0x45394f19c000:[0x4200330c4ddf]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0
2024-08-02T01:50:03.493Z cpu29:2097635)base fs=0x0 gs=0x420047400000 Kgs=0x0
2024-08-02T01:50:02.886Z cpu29:2097635)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x102802(0x420033000000):0x4200349df01f:0xf48] (Src 0x4, CPU29)
2024-08-02T01:50:02.883Z cpu29:2097635)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x102810(0x420033000000):0x4200349df01f:0xf48] (Src 0x4, CPU29)
2024-08-02T01:50:02.878Z cpu29:2097635)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x10283a(0x420033000000):0x4200349df01f:0xf48] (Src 0x4, CPU29)
2024-08-02T01:50:02.861Z cpu29:2097635)NMI: 712: NMI IPI: RIPOFF(base):RBP:CS [0x10283a(0x420033000000):0x4200349df01f:0xf48] (Src 0x4, CPU29)

 

Also, the vmkernel.log is filled with error messages like below:

2024-08-02T01:50:03.747Z cpu3:2098031)pfa_attrconn_add_attributes: failed to add attribute
2024-08-02T01:50:03.747Z cpu3:2098031)pfr_import_tlv_attr_set_value_helper : failed to add attribute value : 50080 (broker4-1.chi2.prod.zpath.net, 1, 0)
2024-08-02T01:50:03.747Z cpu3:2098031)pfa_attrconn_add_attributes: failed to add attribute
2024-08-02T01:50:03.747Z cpu3:2098031)pfr_import_tlv_attr_set_value_helper : failed to add attribute value : 50080 (broker4-1.nyc4.prod.zpath.net, 1, 0)
2024-08-02T01:50:03.747Z cpu3:2098031)pfa_attrconn_add_attributes: failed to add attribute
2024-08-02T01:50:03.747Z cpu3:2098031)pfr_import_tlv_attr_set_value_helper : failed to add attribute value : 50080 (broker4-3.chi2.prod.zpath.net, 1, 0)

 

Environment

Affected NSX versions:

  1. For NSX 3.x, all releases
  2. For NSX 4.x, NSX 4.2.0.1 and all prior releases

Cause

When the vsip module is importing the state associated with a filter, the host will run out of heap memory for the discovered layer7 attributes, then the vmkernel.log is filled with error messages as shown above.  This logging slows down the rest of the vmotion import, causing an NMI PSOD.

Resolution

This issue is resolved in NSX versions 4.2.0.2, 4.2.1, and subsequent releases. To permanently resolve the issue, upgrade to one of the fixed versions.

There is no workaround for this issue.

Powered by Wolken Software