Host experienced a PSOD - Panic Message: @Purplescreen: TSCs are out of sync: cpu68 24862351660613975 > cpu27 92554004271332

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• PSOD with following Backtrace:
  
  
  
  
  
  Version Details: VMware ESXi 6.5.0 build-16576891
  Panic Details: Crash at YYYY-MM-DDT19:25:37.216Z on CPU 27 running world 72316 - NetWorld-VM-XXXXX. VMK Uptime:47345:23:34:26.640
  Panic Message: @BlueScreen: TSCs are out of sync: cpu4 24793438767448724> cpu27 23641111091092
  Backtrace:
  0x439100d1ba80:[0x4180128ee441]PanicvPanicInt@vmkernel#nover+0x545 stack: 0x4180128ee441, 0x0, 0x439100d1bb28, 0x0, 0x23
  0x439100d1bb20:[0x4180128ee4cd]Panic_NoSave@vmkernel#nover+0x4d stack: 0x439100d1bb80, 0x439100d1bb40, 0x0, 0x43, 0x58542d5702f122
  0x439100d1bb80:[0x418012ab6900]Heartbeat_DetectCPULockups@vmkernel#nover+0x458 stack: 0x542d688c115e, 0x680, 0x4301e3806390, 0x418012887d76, 0x98ec91c
  0x439100d1bc00:[0x4180128fe5bc]Timer_BHHandler@vmkernel#nover+0xdc stack: 0xffba3cea09800000, 0x4390ca934000, 0x0, 0x4180128ff16c, 0xef
  0x439100d1bc80:[0x4180128b256f]BH_DrainAndDisableInterrupts@vmkernel#nover+0x9f stack: 0x439100d1bd60, 0xef00000067, 0x0, 0x4180468004c8, 0x417fd2a1bab8
  0x439100d1bd10:[0x4180128d4202]IntrCookie_VmkernelInterrupt@vmkernel#nover+0xc6 stack: 0xef, 0x0, 0x0, 0x41801292fcad, 0x439100d1bdf0
  0x439100d1bd40:[0x41801292fcad]IDT_IntrHandler@vmkernel#nover+0x9d stack: 0x0, 0x41801293e067, 0x4018, 0x4018, 0x0
  0x439100d1bd60:[0x41801293e067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x0, 0x1, 0xfffffffe, 0x418046800000
  0x439100d1be28:[0x418012ac7a36]CpuSchedIdleLoopInt@vmkernel#nover+0x56 stack: 0x418012ac9f31, 0x418046800120, 0x900000216, 0x439100d10020, 0x0
  0x439100d1be98:[0x418012acb61e]CpuSched_IdleLoop@vmkernel#nover+0x16 stack: 0x2, 0x41801287c260, 0x0, 0x0, 0x0
  0x439100d1bea8:[0x4180128d1a37]Init_SlaveIdle@vmkernel#nover+0x7f stack: 0x0, 0x0, 0x0, 0x0, 0x0
  0x439100d1beb8:[0x41801287c260]SMPSlaveIdle@vmkernel#nover+0x24c stack: 0x0, 0x0, 0x0, 0x0, 0x0
  
  
  
  
• In the PSOD Backtrace
  

  #CPU 26's last message before the panic was at YYYY-MM-DDT02:35:15.089Z, however, when it panicked the time is YYYY-MM-DDT##:##:##.###Z:
  #YYYY-MM-DDT02:35:15.089Z cpu26:66488)ScsiDeviceIO: 3035: Cmd(0x43954fc392c0) 0x85, CmdSN 0x7074 from world 68213 to dev "naa.######################################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
  #YYYY-MM-DDT19:25:37.078Z cpu26:65562)World: 2778: PRDA 0x418046800000 ss 0x0 ds 0x4018 es 0x4018 fs 0x4018 gs 0x4018

• Similarly, CPU 27's last log message:
  

  #YYYY-MM-DDT02:05:20.019Z cpu27:66482)nhpsa: hpsa_vmkScsiCmdDone:6531: Sense data: error code: 0x70, key: 0x5, info:00 00 00 00 , cmdInfo:00 00 00 00 , CmdSN: 0x749d, worldId: 0x10a75, Cmd: 0x4d, ASC: 0x20, ASCQ: 0x0

• vs panic timestamp
  

  #YYYY-MM-DDT19:25:37.078Z cpu27:72316)World: 2778: PRDA 0x418046c00000 ss 0x0 ds 0x4018 es 0x4018 fs 0x4018 gs 0x4018

  Note:"ESXi does not write to the TSC registers during runtime. ESXi writes to the TSC registers only in the rare case where the TSCs were already out of sync when the machine powered on (which indicates a hardware or firmware issue), and in that case it just does a one-time resync of the TSCs early in the boot process."

Environment

• VMware vSphere ESXi 7.X
• VMware vSphere ESXI 8.x

Cause

• cpus 26 and 27 have drifted from the others. This is a hardware issue.

Resolution

1. Reach out to the hardware vendor; however the Advanced hardware diagnostic will not show any hardware error, such PSOD are unpredictable and difficult to diagnose.
2. Kindly share the above findings with the hardware vendor and get the respective hardware replaced.