Server 2016 Virtual Machines are experiencing random OS freezing caused by the LDB folder of the Altiris Agent
Article ID: 383226
Updated On:
Products
Issue/Introduction
We would like more information on the contents and purpose of the folder path C:\ProgramData\Symantec\Symantec Agent\Ldb. The only documentation we can find for this path is at KB 150837 , stating that is contains "Agent secure storage files". According to Microsoft's analysis of ProcMon and memory dump data, this folder is involved in some recent freezing we have been experiencing on Windows Server 2016 VMs. We haven't seen the issue occur on any system with the Altiris Agent stopped. None of our 2019 or 2022 servers are impacted - only 2016.
Memory dump analysis is showing involvement of the LDB folder.
Microsoft is also telling us that there's a bug in 2016.
Environment
Agent running on Server 2016 in a virtual machine
Resolution
We have two options that may help with this issue.
1) try changing StorageIOMode to a different value
2) try relocating the Symantec Agent files to another drive, and then making "c:\programdata\symantec\symantec agent" a directory link to that folder.
Option 1:
You can try changing "StorageIOMode" registry value under HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent
The default value is 0, that means files in LDB folder will be accessed in an optimized way from the performance and data integrity point of view.
You may try using the following values:
1 - lower performance but files will not be corrupted in case of power failure.
2 - faster performance but there is a chance files can be corrupted in case of power failure.
SMA restart is needed after the value was added to the registry.
This option was confirmed to resolve the issue for one customer. Either value of 1 or 2 seemed to work for the customer.
Option 2:
Test this process and see if it is helpful to move files to a different drive and create a Directory Link back to the files:
Move folder content from "c:\programdata\symantec\symantec agent" to a folder on another drive i.e. "d:\symantec agent"
Make a directory link to the new location
(i.e. mklink -D "c:\programdata\symantec\symantec agent" "d:\symantec agent")
This will move the LDB files off of C:\ and may mitigate this issue in Server 2016.
These options will hopefully help to work around the bug in Server 2016 which MS is not willing to fix. We would suggest that the customer upgrade to a newer OS version if these options above are not sufficient to work around the MS bug.
Additional Information
ADDITIONAL INFORMATION ABOUT StorageIOMode:
The StorageIOMode registry entry was created purely for performance tuning purposes back in Jan 2014 when LDB storage was introduced in the first place. The problem was that there could be a lot of file operation with LDB files on Package Servers and if Package Server has 50000 packages for example, the file operations take quite a lot of time. The registry changes the method of access to the files, which affects how fast the file operations are. The piece of code that operates on LDB is very stable and very well tested on every OS starting with Windows XP. There has been no new bugs discovered since 2015 I think. The code works with 0 setting on every client machine in the world, including the servers, and it worked on 2016 servers as well until this year.
The root cause of this issue is a Windows problem, which MS has admitted with their Request For Change that was logged in Feb 2024 (change was made in Server 2019+ as Server 2016 is out of mainstream support as of Jan 2022).
Back to the value - it changes how files are written.
0 - The default method. The unbuffered I/O, the method is chosen because it is much faster than #1 and provides better integrity than #2 in our tests. We write data directly to the disk without using Windows cache. That corresponds to Win32 flags FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH. This method works if we can detect the physical sector size, if this detection fails, which it is not in this case w/ 2016 server, then the method switches to #1.
1 - Slowest method. We flush written data every time we write. FILE_FLAG_WRITE_THROUGH is used. This method is not recommended even by MS, since it causes performance penalties.
2 - Fastest performance, Windows cache is used. This is the fastest method but there is a chance of logical LDB corruption in case of unexpected power failure.
You can google these FILE_FLAG_xxx methods, no mystery behind them. They work everywhere except for these 2016 servers.
Feedback
