There are 4 Policy Server but only 3 Policy Servers were in the HCO.
After adding the missing Policy Server in the HCO, agents started throwing HTTP 500 intermittently.
Release : ANY
It is important to check that all Policy Servers are configured the same way.
1. Same encryption keys
2. Same Policy Store
3. Same custom files
4. Same tuning configuration
There was custom java authentication scheme and the problematic policy server was failing to initialize the custom auth scheme due to missing some jar files.
Stop the Policy Server and copy the custom files from a working Policy Server over to this machine.