(-95) Checkpoint did not return status, full response: 'Checkpoint Error: 503 Service Unavailable

Products

• CA App Synthetic Monitor
• DX Application Performance Management

Issue/Introduction

We're currently experiencing an issue where two out of the three OPMS servers are showing the following error. 

(-95) Checkpoint did not return status, full response: 'Checkpoint Error: 503 Service Unavailable. Sorry, cbot agent is too busy at the moment (too many checks pending)'

This issue started happening after servers were restarted during our patching weekend. 

Environment

Release : 10.1

Component : CA APP SYNTHETIC MONITOR (WATCHMOUSE)

Cause

Per Engineering, this seems like some issue AND the HTTP/HTTPS/cbot issue on top of it. There are some dependencies on the Monit's components reporting. Just an example, if API was not running, much of the other stuff would go down, too.

"monit restart fcgiwrap" (as root or with sudo) + wait until all restarted to see.

Resolution

httpbroker is NOT running.

resultbroker is NOT running.

assetsmanager is NOT running.

./smartpop: line 46: /etc/init.d/smartpop-fpm: No such file or directory

browsermob-proxy is running.



To fix it, had to restart each of those processes highlighted and the issue cleared.