VMware ESXi 5.5 Purple Diagnostic Screen Exception 14 in SEsparse and LibAIO (LibAIODrainMergeQueue, LibAIOMergedIODone, SESparseAsyncDataDone)

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on troubleshooting hosts that fail with a purple diagnostic screen referencing Linux native Asynchronous I/O Interface library LibAIO in the stack trace.

Symptoms:

• VMware ESXi 5.5 host fails with a purple diagnostic screen
• You see backtrace similar to:
  
  cpu0:33101)@BlueScreen: #PF Exception 14 in world 33101:memMap-0 IP 0x4180182f4948 addr 0x4108fffffff0 PTEs:0x100088063;0x80000020ad5bf063;0x0;
  cpu0:33101)Code start: 0x418018000000 VMK uptime: 1:09:27:02.593
  cpu0:33101)0x4123c535cb20:[0x4180182f4948]LibAIODrainMergeQueue@vmkernel#nover+0x150 stack: 0x4130002a85c0
  cpu0:33101)0x4123c535cb80:[0x4180182f53fd]LibAIOMergedIODone@vmkernel#nover+0x211 stack: 0x412ec622ef90
  cpu0:33101)0x4123c535cbb0:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x1
  cpu0:33101)0x4123c535cbe0:[0x418018bb9798]SESparseAsyncDataDone@esx#nover+0x15c stack: 0x41300007b0c0
  cpu0:33101)0x4123c535cc10:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x4123c535cc70
  cpu0:33101)0x4123c535cc40:[0x4180182cdc22]FS_IOAccessDone@vmkernel#nover+0x56 stack: 0x412ec043e340
  cpu0:33101)0x4123c535cc70:[0x41801802d21f]AsyncPopCallbackFrameInt@vmkernel#nover+0xe7 stack: 0x4123c535cc90
  cpu0:33101)0x4123c535cca0:[0x4180182fd5e1]FDSAsyncTokenIODone@vmkernel#nover+0xdd stack: 0x4123c535cdb0
  cpu0:33101)0x4123c535cda0:[0x418018320593]SCSICompleteDeviceCommand@vmkernel#nover+0x46f stack: 0x4123c535cdc0
  cpu0:33101)0x4123c535cdd0:[0x4180187dbebe][email protected]#v2_2_0_0+0x82 stack:
  cpu0:33101)0x4123c535cee0:[0x4180187dc18e][email protected]#v2_2_0_0+0x12a stack: 0
  cpu0:33101)0x4123c535cf60:[0x41801834153a]SCSICompletePathCommand@vmkernel#nover+0x292 stack: 0xe958503533d2
  cpu0:33101)0x4123c535cfd0:[0x41801832eaf3]SCSICompleteAdapterCommand@vmkernel#nover+0x13f stack: 0x4180184a69f
  cpu0:33101)0x4123c535d0f0:[0x4180186d9c13][email protected]#9.2+0x3e3 stack: 0x4123c535
  cpu0:33101)0x4123c535d250:[0x4180180eb521]WorldletProcessQueue@vmkernel#nover+0x779 stack: 0x417f00000000
  cpu0:33101)0x4123c535d290:[0x4180180eb93c]WorldletBHHandler@vmkernel#nover+0x54 stack: 0x4123c535d2d0
  cpu0:33101)0x4123c535d320:[0x41801802e94f]BH_DrainAndDisableInterrupts@vmkernel#nover+0xf3 stack: 0x4123c535d4
  cpu0:33101)0x4123c535d360:[0x418018063e83]IDT_IntrHandler@vmkernel#nover+0x1af stack: 0x4123c535d480
  cpu0:33101)0x4123c535d370:[0x4180180f1064]gate_entry@vmkernel#nover+0x64 stack: 0x4018
  cpu0:33101)0x4123c535d480:[0x4180183a329a]Power_HaltPCPU@vmkernel#nover+0x1fe stack: 0x0
  cpu0:33101)0x4123c535d4f0:[0x41801824e859]CpuSchedIdleLoopInt@vmkernel#nover+0x4bd stack: 0x4123c535d580
  cpu0:33101)0x4123c535d650:[0x418018254930]CpuSchedDispatch@vmkernel#nover+0x1630 stack: 0x0
  cpu0:33101)0x4123c535d6c0:[0x418018255c65]CpuSchedWait@vmkernel#nover+0x245 stack: 0x1
  cpu0:33101)0x4123c535d740:[0x4180182565c4]CpuSched_TimedWait@vmkernel#nover+0xec stack: 0x0
  cpu0:33101)base fs=0x0 gs=0x418040000000 Kgs=0x0
   
• You observe this issue on a View environment with ESXi 5.5 using SESparse or Reclaim disk space.

Environment

VMware vSphere ESXi 5.5

Cause

This issue occurs due to a new feature introduced in the AIO Library to improve SESparse performance and may occur when using SESparse snapshots.

Resolution

This issue is resolved in ESXi 5.5 Patch Release ESXi550-201505002. For more information, see VMware ESXi 5.5, Patch Release ESXi550-201505002 (2114245).

To work around this issue, you must disable LibAIO coalesce Read and Write.

To disable LibAIO coalesce Read and Write:

 

1. Run this command on the affected ESXi host to verify current settings for coalesce Read and Write:
    

   esxcfg-advcfg -g /FDS/FDSEnableCoalesceWrites

   esxcfg-advcfg -g /FDS/FDSEnableCoalesceReads

   
   Note: Value 1 indicates Enabled and value 0 indicates Disabled.
2. If you see the value 1, run this command on the ESXi host to disable the setting:
   
   esxcfg-advcfg -s 0 /FDS/FDSEnableCoalesceWrites
   esxcfg-advcfg -s 0 /FDS/FDSEnableCoalesceReads
   
   Note: Changes to these advanced parameters can be changed in a live system and they are persistent.

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box
VMware ESXi 5.5 で SESparse と LibAIO (SESparseAsyncDataDone、LibAIODrainMergeQueue、LibAIOMergedIODone) において紫色の診断画面に Exception 14 が表示される
SEsparse 和 LibAIO 中 VMware ESXi 5.5 紫色诊断屏幕异常 14（LibAIODrainMergeQueue、LibAIOMergedIODone、SESparseAsyncDataDone）
VMware ESXi 5.5, Patch Release ESXi550-201505002