PSOD with pfa_value2str

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x
• VMware vDefend Firewall

Issue/Introduction

• ESXi may encounter PSOD with below backtrace:

yyyy-mm-ddT09:00:46.647Z cpu1:145731248)World: 3072: PRDA 0x420040400000 ss 0x0 ds 0xf50 es 0xf50 fs 0x0 gs 0x0
yyyy-mm-ddT09:00:46.647Z cpu1:145731248)World: 3074: TR 0xf68 GDT 0xfffffffffca02000 (0xffff) IDT 0xfffffffffc408000 (0xffff)
yyyy-mm-ddT09:00:46.647Z cpu1:145731248)World: 3075: CR0 0x80050033 CR3 0x1d84f7000 CR4 0x152660
yyyy-mm-ddT09:00:46.696Z cpu1:145731248)Backtrace for current CPU #1, worldID=145731248, fp=0x4539c5c19720
yyyy-mm-ddT09:00:46.696Z cpu1:145731248)0x4539c5c180a0:[0x4200225441ab][email protected]#1.0.7.0.24150844+0x7 stack: 0x4539c5c19720, 0x420022549be1, 0x202020534e, 0x0, 0x0
yyyy-mm-ddT09:00:46.726Z cpu1:145731248)ESC[45mESC[33;1mVMware ESXi 7.0.3 [Releasebuild-23794027 x86_64]ESC[0m
#PF Exception 14 in world 145731248:DVFilter-Asy IP ######### addr #####
PTEs:0x0;
yyyy-mm-ddT09:00:46.738Z cpu1:145731248)cr0=0x8001003d cr2=0x7 cr3=0x8cd02000 cr4=0x10216c
yyyy-mm-ddT09:00:46.741Z cpu1:145731248)FMS=06/55/4 uCode=0x2007108
yyyy-mm-ddT09:00:46.746Z cpu1:145731248)frame=0x4539c5c17fe0 ip=0x4200225441ab err=0x0 rflags=0x10206
yyyy-mm-ddT09:00:46.748Z cpu1:145731248)rax=0x0 rbx=0x4 rcx=0x0
yyyy-mm-ddT09:00:46.753Z cpu1:145731248)rdx=0x442 rbp=0x4539c5c19720 rsi=0x4539c5c19720
yyyy-mm-ddT09:00:46.756Z cpu1:145731248)rdi=0x5 r8=0x117a r9=0x41ffe18039a0
yyyy-mm-ddT09:00:46.760Z cpu1:145731248)r10=0x0 r11=0x4334eb200e58 r12=0x4539c5c19720
yyyy-mm-ddT09:00:46.765Z cpu1:145731248)r13=0x4539c5c199c8 r14=0x4539c5c18618 r15=0x4539c5c19a40
yyyy-mm-ddT09:00:46.768Z cpu1:145731248)pcpu:0 world:227136232 name:"vmm1:xxxx" (V)
yyyy-mm-ddT09:00:46.769Z cpu1:145731248)pcpu:1 world:145731248 name:"DVFilter-Async-3" (S)
yyyy-mm-ddT09:00:46.769Z cpu1:145731248)pcpu:2 world:227195037 name:"vmm0:TEPI" (V)
yyyy-mm-ddT09:00:46.769Z cpu1:145731248)pcpu:3 world:227209107 name:"vmm0:xxxxx" (V)
yyyy-mm-ddT09:00:46.769Z cpu1:145731248)pcpu:4 world:222332095 name:"vmm1:xxxxx" (V)
yyyy-mm-ddT09:00:46.769Z cpu1:145731248)pcpu:5 world:227193890 name:"vmm1:xxxxx" (V)
yyyy-mm-ddT09:00:46.770Z cpu1:145731248)pcpu:6 world:145731250 name:"vdpi" (U)

• PSOD screenshot reference:

Environment

ESXi 7.0 or ESXi 8.0 which prepared as NSX 4.1.X (or earlier version) transport node

Cause

According to the memory dump, the PSOD happened because the thread accessed attribute_log->log_value[4], which exceeded the array size and resulted in invalid memory access.
The issue happens while L7 distribute firewall been configured with related context profiles, The problem is in the attribute_match_helper functions, which would increment the num_values even if there was no space to store a new attribute.

Resolution

Issue is fixed in NSX T 4.2.0 and later version

Workaround:
- To change the L7 firewall rules to L4 firewall rules, if possible.