Policy Server generating keys was set to roll over Agent Keys every Monday morning at a fixed time. Errors were occurring on Web Agents that were configured to talk to the Policy Servers that were not generating keys.
Web Agents report errors:
HTTP 500 server error '10-0004'.
[8599/3086460608][Wed Dec 03 2008 14:01:21][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.
[8598/3086460608][Wed Dec 03 2008 14:01:21][CSmHttpPlugin.cpp:274][ERROR] Unable to resolve server host name.
Exiting with HTTP 500 server error '10-0004'.
This would cause the login fcc server to throw errors too:
[Communication failure between SiteMinder policy server and web agent.][GET]
[CSmProtectionManager::DoIsProtected][LowLevelAgent returned SmFailure.][GET]
[ProcessAdvancedAuthentication][ProtectionManager returned SmNoAction or SmFailure, end new request.][GET]
Agent key updates from the key generating Policy Servers via servercommands were not reaching the Policy Servers in the data center in the other geographical zone.
IMPORTANT: This article contains information about modifying the registry.
Before modifying the registry, make sure to create a backup of the registry and ensure to understand how to restore the registry if a problem may occur.
For more information about how to back up, restore, and edit the registry, please review the relevant Microsoft Knowledge Base articles on support.microsoft.com.
The reason for this issue was:
Policy servers that were not generating keys to the shared Key Store did not have the registry setting EnableKeyUpdate set to 1.
When a single Policy Server generates encryption keys in an environment with multiple Policy Servers that connect to disparate Policy Stores but share a central Key Store, an additional registry setting is required. This registry setting configures each Policy Server to poll the common Key Store and retrieve new encryption keys at a regular interval.
EnableKeyUpdate= 1; REG_DWORD
Once this registry setting is enabled on all the Policy Servers except the one which generates the Keys, all the Policy Servers are set to receive the keys from the new key generated by Policy Server. This solved the issue.