SiteMinder policy server that was generating keys was set to rollover agent keys every Monday morning at a fixed time. Errors were occurring on WebAgents that were configured to talk to the policy servers that were not generating keys.
Errors on WebAgents:
HTTP 500 server error '10-0004'.
[8599/3086460608][Wed Dec 03 2008 14:01:21][CSmResourceManager.cpp:155][WARNING] HLA: Missing resource data.
[8598/3086460608][Wed Dec 03 2008 14:01:21][CSmHttpPlugin.cpp:274][ERROR] Unable to resolve server host name. Exiting with HTTP 500 server error '10-0004'.
This would cause the login fcc server to throw errors too:
[12/02/2008][22:00:40][03f40d54-6a1f-4935f658-003c-274077eb][IsResourceProtected] [Communication failure between SiteMinder policy server and web agent.][GET]
[12/02/2008][22:00:40][03f40d54-6a1f-4935f658-003c-274077eb] [CSmProtectionManager::DoIsProtected][LowLevelAgent returned SmFailure.][GET]
[12/02/2008][22:00:40][03f40d54-6a1f-4935f658-003c-274077eb] [ProcessAdvancedAuthentication][ProtectionManager returned SmNoAction or SmFailure, end new request.][GET]
Cause of this issue:-
Agent key updates from the key generating policy servers via servercommands were not reaching the policy servers in the data center in the other geographical zone.
IMPORTANT: This article contains information about modifying the registry.
Before you modify the registry, make sure to create back up of the registry and ensure that you understand how to restore the registry if a problem may occur.
For more information about how to back up, restore, and edit the registry, please review the relevant Microsoft Knowledge Base articles on support.microsoft.com.
This reason for this issue was -Policy servers that were not generating keys to the shared key store did not have the registry setting EnableKeyUpdate set to 1.
When a single Policy Server generates encryption keys in an environment with multiple Policy Servers that connect to disparate policy stores, but share a central key store, an additional registry setting is required. This registry setting configures each Policy Server to poll the common key store and retrieve new encryption keys at a regular interval.
HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\ObjectStore=26 966 EnableKeyUpdate= 1; REG_DWORD
Once this registry SiteMinder registry setting was enabled on the policy server set to receive the keys from the key generating policy servers, The issue was resolved.