Windows VM Crash During High I/O – Cause Traced to cysvc!CySvcServiceHandler (Palo Alto Cortex)
Article ID: 405757
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Windows Server virtual machines experienced intermittent unresponsiveness and crashes, particularly during high I/O operations.
VMware logs did not reveal any hypervisor-level backtraces or failures. The root cause was identified through analysis of Windows crash dumps.
Cause
Windows Crash dump analysis revealed that the system halted during critical file system cache flush and write operations (Ntfs!, CcFlushCache).
The issue with the cysvc!CySvcServiceHandler module, indicating that Palo Alto Cortex (formerly Cylance) was unable to handle I/O requests under high load.
This resulted in a kernel-level failure or deadlock within the Windows guest operating system.
Resolution
-
Uninstall/disable Palo Alto’s Cortex endpoint protection software from the affected VM to isolate the issue.
-
After uninstallation, the abnormal behavior (unresponsiveness and crashes during high I/O operations) was no longer observed.
-
If similar symptoms recur, contact Palo Alto support to:
-
Investigate potential driver-level issues
-
Provide updated drivers or patches
-
Review and adjust Cortex configuration settings
-
Additional Information
Windows crash dump analysis
Back trace
# Child-SP Return Call Site Info
0 ffffdd8e0645d210 fffff80286035090 nt!KiSwapContext+0x76
1 ffffdd8e0645d350 fffff802860d03b1 nt!KiSwapThread+0x6a0
2 ffffdd8e0645d420 fffff802860cf081 nt!KiCommitThreadWait+0x271
3 ffffdd8e0645d4c0 fffff802192f8ef2 nt!KeWaitForSingleObject+0x6e1
4 ffffdd8e0645d5b0 fffff80219411c59 Ntfs!NtfsWaitOnIo+0x82
5 ffffdd8e0645d610 fffff802192fafe1 Ntfs!NtfsNonCachedIo+0x869
6 ffffdd8e0645d7b0 fffff802193195ca Ntfs!NtfsNonCachedUsaWrite+0xe1
7 ffffdd8e0645d870 fffff8021931b094 Ntfs!NtfsCommonWrite+0x36fa
8 ffffdd8e0645da60 fffff802860987ed Ntfs!NtfsFsdWrite+0x584
9 (Inline) ---------------- nt!IopfCallDriver+0xb5
a ffffdd8e0645dbd0 fffff80217a6cfb5 nt!IofCallDriver+0xcd
b ffffdd8e0645dc10 fffff80217a6c095 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x255
c ffffdd8e0645dca0 fffff802860987ed FLTMGR!FltpDispatch+0x105
d (Inline) ---------------- nt!IopfCallDriver+0xb5
e ffffdd8e0645dd40 fffff80286097d39 nt!IofCallDriver+0xcd
f ffffdd8e0645dd80 fffff802860970ae nt!IoSynchronousPageWriteEx+0x159
10 ffffdd8e0645ddc0 fffff802192fea93 nt!IoSynchronousPageWrite+0x1e
11 ffffdd8e0645de10 fffff8021931b286 Ntfs!LfsSynchronousPageWrite+0x73
12 ffffdd8e0645deb0 fffff8021930067c Ntfs!LfsFlushHeadOfTheLog+0x36
13 ffffdd8e0645def0 fffff802192ffa55 Ntfs!LfsFlushLfcb+0xc0c
14 ffffdd8e0645e0f0 fffff802860136d4 Ntfs!LfsFlushLfcbCallout+0x25
15 ffffdd8e0645e120 fffff802860135ed nt!KeExpandKernelStackAndCalloutInternal+0xd4
16 ffffdd8e0645e190 fffff802192ff6ca nt!KeExpandKernelStackAndCalloutEx+0x1d
17 ffffdd8e0645e1d0 fffff802194d1b80 Ntfs!LfsFlushLfcbOnNewStack+0x5a
18 ffffdd8e0645e230 fffff8021941f89c Ntfs!LfsFlushToLsnPriv+0x160
19 ffffdd8e0645e2c0 fffff802193196ea Ntfs!LfsFlushToLsnWithoutDiskCacheFlush+0xac
1a ffffdd8e0645e310 fffff8021931b094 Ntfs!NtfsCommonWrite+0x381a
1b ffffdd8e0645e500 fffff802860987ed Ntfs!NtfsFsdWrite+0x584
1c (Inline) ---------------- nt!IopfCallDriver+0xb5
1d ffffdd8e0645e670 fffff80217a6cfb5 nt!IofCallDriver+0xcd
1e ffffdd8e0645e6b0 fffff80217a6c095 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x255
1f ffffdd8e0645e740 fffff802860987ed FLTMGR!FltpDispatch+0x105
20 (Inline) ---------------- nt!IopfCallDriver+0xb5
21 ffffdd8e0645e7e0 fffff80286097a89 nt!IofCallDriver+0xcd
22 (Inline) ---------------- nt!IoSynchronousPageWriteEx+0x13b
23 (Inline) ---------------- nt!MiSynchronousPageWrite+0x13b
24 ffffdd8e0645e820 fffff8028606de15 nt!MiIssueSynchronousFlush+0x181
25 ffffdd8e0645e890 fffff8028619aa42 nt!MiFlushSection+0x635
26 ffffdd8e0645eaa0 fffff8028619a088 nt!MmFlushSection+0x142
27 ffffdd8e0645eb60 fffff8028609e6f7 nt!CcFlushCacheOneRange+0x338
28 ffffdd8e0645ec50 fffff80286294170 nt!CcFlushCachePriv+0x137
29 ffffdd8e0645ecd0 fffff80219401abe nt!CcCoherencyFlushAndPurgeCache+0xa0
2a (Inline) ---------------- Ntfs!NtfsCoherencyFlushAndPurgeCache+0x37
2b ffffdd8e0645ed40 fffff80219563a1d Ntfs!NtfsFlushUserStream+0x14e
2c ffffdd8e0645edf0 fffff802194fa719 Ntfs!NtfsPerformOptimisticFlush+0xa9
2d ffffdd8e0645ee40 fffff802194fa0a9 Ntfs!NtfsCommonFlushBuffers+0x5d9
2e ffffdd8e0645ef50 fffff802860136d4 Ntfs!NtfsCommonFlushBuffersCallout+0x19
2f ffffdd8e0645ef80 fffff802860135ed nt!KeExpandKernelStackAndCalloutInternal+0xd4
30 ffffdd8e0645eff0 fffff802194e600c nt!KeExpandKernelStackAndCalloutEx+0x1d
31 (Inline) ---------------- Ntfs!NtfsCommonFlushBuffersOnNewStack+0x5d
32 ffffdd8e0645f030 fffff802860987ed Ntfs!NtfsFsdFlushBuffers+0x15c
33 (Inline) ---------------- nt!IopfCallDriver+0xb5
34 ffffdd8e0645f100 fffff80217a6cfb5 nt!IofCallDriver+0xcd
35 ffffdd8e0645f140 fffff80217a6c095 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x255
36 ffffdd8e0645f1d0 fffff802860987ed FLTMGR!FltpDispatch+0x105
37 (Inline) ---------------- nt!IopfCallDriver+0xb5
38 ffffdd8e0645f270 fffff802866d33a8 nt!IofCallDriver+0xcd
39 (Inline) ---------------- nt!IoCallDriverWithTracing+0x25
3a (Inline) ---------------- nt!IopCallDriverReference+0xad
3b ffffdd8e0645f2b0 fffff8028672fa94 nt!IopSynchronousServiceTail+0x1c8
3c ffffdd8e0645f360 fffff8028672f83c nt!IopFlushBuffersFile+0x240
3d ffffdd8e0645f400 fffff8028672f7a6 nt!NtFlushBuffersFileEx+0x7c
3e ffffdd8e0645f460 fffff802864b8541 nt!NtFlushBuffersFile+0x16
3f ffffdd8e0645f4a0 00007fff21242464 nt!KiSystemServiceExitPico+0x496
40 000000dd13eff438 00007fff20b03fc9 ntdll!ZwFlushBuffersFile+0x14
41 000000dd13eff440 00007fff07aa3cf8 KernelBase!FlushFileBuffers+0x29
42 000000dd13eff480 0000000000000000 cysvc!CySvcServiceHandler+0x168f438Feedback
Yes
No
Powered by
