Having multiple messages in the agent logs about the virtual directories are creating/updating.

Products

IT Management Suite Client Management Suite

Issue/Introduction

After upgrading to ITMS 8.5 RU4 (from 8.0), the customer started having multiple messages in the agent logs about the virtual directories are creating/updating.

During this process of Creating/updating virtual directory activity, no PackageDownload events happen. Then, as soon as that stops, the packages start downloading again. When a new package comes down to the Package Server (PS), the customer sees Creating/updating virtual directories.

The Creating/updating virtual directory can take multiple hours and that stops all package downloads. This is causing an issue in the customer's environment.

Also, it was noticed some slowness in part because of the continual hash checking of each file and the customer has some packages with thousands of files and thousands of folders.

Note: The customer uses custom UNC folders/shares to store the files for their packages.

Questions:

Why we continually see Creating/updating virtual directories?
Is there a way to change the hash checking to how it was before, set it at a higher (folder) level, or disable it entirely?

Environment

ITMS 8.5, 8.6

Cause

While Creating/updating virtual directory messages occurs, nothing is stopped. Those processes run in different threads.
Downloading is done by Package Delivery in parallel with Package Server main thread which is updating virtual directories.

Of course, there is a dependency between Creating/updating virtual directory and downloading new packages, since Package Server (PS) thread is triggering the package download and putting packaged to the queue in Package Delivery (which in own turn start to download packages from its own queue). So if Package Server (PS) is busy recreating virtual directories, then it could not trigger a download for some period.

We have changed the approach of creating virtual directories and use the native IIS API with our 8.5 and later. However, having multiple custom shared folders hosting packages can add extra work to this process.

Previously the Package Server (PS) was working through IIS6 compatibility mode, using IIS "Metabase" and related binaries. Starting IIS7, the officially proper approach is to modify web.config files and "applicationhost.config" files to manage virtual infrastructure of IIS. The Symantec Management Agent (the Package Server specifically) was ported to use that approach. The logic of when the virtual directories are created wasn't changed, only the way how they are now changed.

In the situation when there are thousands of virtual directories, those config files grow quite big and the IIS engine spends a lot of time parsing those and making changes in those. The more packages bring more slowness, that is why the approach to have a separate directory for each package is not recommended and discouraged.

Resolution

Answers:

Why we continually see Creating/updating virtual directories?

Virtual directories are (re)created when some custom package is downloaded\updated. Then the virtual directory gets to the cache list in PS memory. Each time when an updated policy is received the virtual directories are checked/verified with that cache list. That is when "Creating/updating" comes from. In environments where custom folders hosting multiple packages are used (one folder per package) can cause issues. For example, if you have 3000 custom folders with files, it means each time there is a need to run an update for 3K+ virtual directories. As we said before, their problem is in the approach. The packages should be stored in the main PS storage and not custom shared locations. The custom path packages are exceptions from the rule, but the customer uses this as a main line approach, which is not the correct usage of the feature. When packages are stored in PS storage, then there are only one or two virtual directories, not 3K+ for example.

More custom packages the customer adds - slower it will be.

In 8.5 and later, we have added the unconditional informational log message "Creating/Updating ...". Previously, it was logged only in case of the creation of a virtual directory and the update process was not reflected in the log. But this is only about logging a message, the update virtual directories logic is running in both cases and based on the same conditions.

The process of creation of each virtual directory is divided in two phases: search for virtual directory, if not exist - create it, and if it exists - then update virtual directory details.
In previous 8.0 logic, the message was only while the creation of the directory and update was done quietly, so we were having a hard time sometimes seeing from the logs what directories actually exist, whether they were updated, etc.
Is there a way to change the hash checking to how it was before, set it at a higher (folder) level, or disable it entirely?

The hashes of files are checked when Package Server (PS) is started. It checks hashes of package files from disk and caches this in memory. Then on the next "package refresh" cycles, it checks hashes only if a file is changed or metadata is changed (new snapshot, etc). and use cached flags about file hashes and do not re-hash them on every cycle, since it is not cheap.
This process has nothing to do with virtual directories on IIS. Also, there is no such thing as a hash check on folder level. We check hashes only for files.

You could try enabling the legacy way regarding virtual directories. In the registry "Software\Altiris\Altiris Agent\Package Server" it is possible to switch to the legacy IIS API using DWORD key "UseLegacyIISAPI" which should be greater than zero. Although, the admin then should be sure it has IIS6 compatibility features installed for IIS (SMA does not do this check anymore). However, since on latest IIS versions the "IIS6 Compatibility" is an additional wrapper, which at the end also uses the native IIS7 API.

Starting with ITMS 8.6 RU1, Two registry settings were added:

"IISVirtualDirectoriesRefreshInterval (min)" and "IISVirtualDirectoriesLastRefreshTime"

These registries will control whether refresh should be run now or could be skipped. By default the refresh, if required at all, will be run not often than once in 24 hours. The "IISVirtualDirectoriesRefreshInterval" for example is created when it is asked by the code for the first time. The registry keys are created automatically with correct names, even if someone has created the wrong named keys. No agent service restart is required.

Be aware, that after Package Server restart, there will be one cycle of refresh, both virtual directories and packages. In case when refresh is skipped, there will be an INFO level log message: "Skip virtual directories and file shares refresh. Last refresh: <datetime>".

Also, these two registries do not turn off all refreshes. It only turns off exceeded refreshes based on a cached list. The other situations which lead to the refresh are still working.

When "Creating/updating" goes in pair with "Refreshing" means the refresh process is done on a per-package level. These two registries avoid extra virtual directories refresh based on a cached list of virtual directories. But there are also a mandatory refresh processes (like user click "Resend all statuses") or in case of some major change in policy that affects all packages (HTTP/UNC change, credentials change, storage moved, etc). Then refresh goes package by package and updates all the required stuff regarding the package, including virtual directory.

In order to see that the change is working, check the log for message: "Skip virtual directories and file shares refresh." If it is there, then these two registries are working.

FYI: Package Server full package refresh is also regulated in the registry by keys: "LastPackageRefreshTime" and "AgentExecInterval (sec)" under HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Package Server. If a policy is changed and the "full refresh time" + "refresh interval" is exceeded, then full refresh will be done.
"AgentExecInterval (sec): 86400" this equals to 24 hours. Means the "Full Package Server refresh" will be run every 24 hours.
"AgentExecInterval (sec): 604800" this equals to 24 hours. Means the "Full Package Server refresh" will be run once a week.

Additional Information

KB 210634 "Reoccurring Messages About Creating/updating virtual directories"