Even though all upgradeable discovery_agents were upgraded customer was getting alarms as if the discovery_server doesn't think the discovery_agent's were actually upgraded, and this occurred despite discovery_agent cold start/restart.
".../discovery_agent with incompatible version 6.50 : Please update to discovery_agent version 7.00 or later."
alarm example:
Oct 10 10:17:30:484 ERROR [nimclient, discovery_server] Exception in ThreadClient: (2) communication error, I/O error on nim session (S) com.nimsoft.nimbus.NimServerSession(/172.30.33.150:48032-/172.30.33.150:64891): Connection reset
and
Oct 10 10:17:30:484 ERROR [nimclient, discovery_server] (2) communication error, I/O error on nim session (S) com.nimsoft.nimbus.NimServerSession(/172.30.xx.xxx:xxxxx-/172.xx.x.xxx:xxxxx): Connection reset
Then we saw exception in thread client errors in the log as per above example.
Note that the new discovery_agent v7.0 is not supported on 32-bit OSs, you need a 64-bit OS.
Even after discovery_server cold start we received the same communication issues/connect reset/thread client errors.
Resolution:
1) Using the Infrastructure Manager (IM), Deactivate existing discovery_agent probe v7.0
2) Delete the discovery_agent v7.0 files/folder from the file system (note discovery Scopes were not deleted/affected)
3) Redeploy the discovery_agent probe v7.0
Then the alarms were eliminated and did not return.