VM vmx Crash fails with mks - PANIC: VERIFY bora/vmx/main/pollVMX.c:4348
Article ID: 394512
Updated On:
Products
Issue/Introduction
VM crashes and we may see events similar to these under VM > Monitor > Events:
"A ticket of type VirtualMachine.TicketType.webmks.label has been acquired"
vmware.log:
2025-03-24T12:36:35.554Z In(05) vmx - VigorTransport_ServerSendResponse opID=31c6da20-3f-74c5 seq=949803: Completed MKS.IssueTicket request with messages in 102 US.
2025-03-24T12:36:35.664Z No(00) vmx - <<< file Throttled >>>
2025-03-24T12:37:41.173Z Cr(01) mks - PANIC: VERIFY bora/vmx/main/pollVMX.c:4348
2025-03-24T12:37:42.021Z Wa(03) mks - A core file is available in "/vmfs/volumes/<datastore_UUID>/<vm_name>/vmx-zdump.000"
2025-03-24T12:37:42.021Z In(05) mks - Backtrace:
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[0] 000000866882cbf0 rip=0000008623d62e30 rbx=0000000000000001 rbp=000000866882d0f0 r12=00000086252044a8 r13=000000866882cc10 r14=0000000000000000 r15=0000000000001848
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[1] 000000866882d100 rip=0000008623d57911 rbx=0000000000000004 rbp=000000866882d150 r12=0000008667823010 r13=0000000000001844 r14=0000000000000000 r15=0000000000001848
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[2] 000000866882d160 rip=0000008623d58855 rbx=000000000000c220 rbp=000000866882d1f0 r12=000000866782f230 r13=0000000000000004 r14=000000866882d400 r15=0000008667823010
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[3] 000000866882d200 rip=0000008623e5cfe9 rbx=00000086252231c0 rbp=000000866882d210 r12=00000086257a7830 r13=0000037eef7cf84f r14=000000866882d400 r15=0000000000001000
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[4] 000000866882d220 rip=000000862454c7e9 rbx=0000000000000000 rbp=000000866882d340 r12=00000086257a7830 r13=0000037eef7cf84f r14=000000866882d400 r15=0000000000001000
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[5] 000000866882d350 rip=000000866644b852 rbx=0000000000000000 rbp=0000000000000000 r12=0000037eef7cf84e r13=0000037eef7cf84f r14=000000866882d400 r15=0000000000001000
Core Dump:
(gdb) bt
#0 0x00000086245fed4d in VMKernel_LiveCoreDump (coreFileNameLen=4196, coreFileName=0x866882bb40 "") at bora/vmkernel/public/x86/uwvmk.h:756
#1 Sig_CoreDump () at bora/lib/sig/sigPosix.c:2326
#2 0x0000008623d62e96 in Panic (fmt=fmt@entry=0x8624911f4a "VERIFY %s:%d\n") at bora/vmx/main/vmx.c:1676
#3 0x0000008623d57911 in PollAddDevice (e=<optimized out>, class=<optimized out>, poll=<optimized out>) at bora/vmx/main/pollVMX.c:4348
#4 PollUpdateState (poll=poll@entry=0x8667823010, callerClass=callerClass@entry=POLL_CLASS_MKS) at bora/vmx/main/pollVMX.c:2173
#5 0x0000008623d58855 in PollVMXLoopTimeout (loop=0 '\000', exit=0x86252231c0 <mks> "", class=<optimized out>, timeout=1000000) at bora/vmx/main/pollVMX.c:1992
#6 0x0000008623e5cfe9 in MKSThreadBase (data=<optimized out>) at bora/mks/main/mksThread.c:915
#7 0x000000862454c7e9 in VThreadThreadWrapper (clientData=0x0) at bora/lib/thread/vthreadPosix.c:406
#8 0x000000866644b852 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x000000866655923f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[6] 000000866882d400 rip=000000866655923f rbx=0000008668831700 rbp=0000000000000000 r12=0000037eef7cf84e r13=0000037eef7cf84f r14=000000866882d400 r15=0000000000001000
2025-03-24T12:37:42.021Z In(05) mks - Backtrace[7] 000000866882d408 rip=0000000000000000 rbx=0000008668831700 rbp=0000000000000000 r12=0000037eef7cf84e r13=0000037eef7cf84f r14=000000866882d400 r15=0000000000001000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[0] 000000866882cbf0 rip=0000008623d62e30 in function Panic in object /bin/vmx loaded at 0000008623acc000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[1] 000000866882d100 rip=0000008623d57911 in function (null) in object /bin/vmx loaded at 0000008623acc000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[2] 000000866882d160 rip=0000008623d58855 in function (null) in object /bin/vmx loaded at 0000008623acc000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[3] 000000866882d200 rip=0000008623e5cfe9 in function (null) in object /bin/vmx loaded at 0000008623acc000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[4] 000000866882d220 rip=000000862454c7e9 in function (null) in object /bin/vmx loaded at 0000008623acc000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[5] 000000866882d350 rip=000000866644b852 in function (null) in object /lib64/libpthread.so.0 loaded at 0000008666444000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[6] 000000866882d400 rip=000000866655923f in function clone in object /lib64/libc.so.6 loaded at 0000008666464000
2025-03-24T12:37:42.022Z In(05) mks - SymBacktrace[7] 000000866882d408 rip=0000000000000000
Environment
- vSphere ESXi 8.x
- vSphere ESXi 7.x
- VMware Remote Console 11.x, 12.x (all supported version)
Cause
The VM is getting spammed with console ticket requests due to a faulty VMRC build interacting poorly with a proxy or SSL error. This provokes an ESX bug where too many outstanding console tickets would cause a crash.
Resolution
This has be resolved in VMware ESX 9.0 and VMware Remote Console 13.0.
Workaround:
Set the this VM config option:
-We can set this in the VM's VMX file directly or from the Host Client UI:
VM > Edit > Advanced > Edit Configuration
ticket.webmks.timeout.seconds = 10
The triggering factor here is normally something funny about SSL certificates or an intermediate proxy that's misbehaving, so you may also be able to work-around it by fixing an underlying certificate/proxy issue that caused the initial connection failure.
- Or we can set this in /etc/vmware/config on a specific host for changing all VMs on a host.
# vi /etc/vmware/config (and add the following line in this fifle)
ticket.webmks.timeout.seconds = 10
and need to restart the VM (power off / on a VM or vmotion to this host or another host which have also the same config)
Feedback
