When running Policy Server, and since two days, heavy slowness is seen on some of the Policy servers. At the time of the issue, messages like this one show up in the Policy Server log:
Connection request rejected. Connection limit of 3072 exceeded
[Sm_Auth_Message.cpp:511][INFO][sm-log-00000] Execution time exceeded threshold. (CSm_Auth_Message::ProcessMessage, 19953, 5000, agent=<agent> client=*10.0.0.1 server=https://myserver.example.com resource=/index.html action=POST user=)
The available file descriptors (nofiles) for the Policy Server process, 1024 is too low. Setting the Max Connection to 3072 will lead to request more than 1027 available file descriptors. The "available file descriptors" (nofiles) should be configured at the OS level (1).
Set the value of ulimit -n to a value much higher than 1024 to fix this issue (1).