Remote collector(Cloud proxy) is in down status even there is no network issue and not recovered
Article ID: 319795
Updated On:
Products
Issue/Introduction
For explain this issue for checking problem easily.
Symptoms:
Remote collector(Cloud proxy) is in down status even there is no network or firewall issue and not recovered.
Remote collectors are running as cloud proxy after upgrading above 8.10 release.
Reboot remote collector(CP) does not recovered this issue.
You can see the log entries like below from collector.log
2023-09-12T04:02:25,046+0000 WARN [poolTimer-CollectorAnalyticsPool-3 tid=50 ] org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection - Could not create a new connection to server: 172.18.68.151(vRealize Ops Analytics-08d7bd38-ffb7-4a58-8827-2c54b14c9037:23788)<ec><v4>:10002 refused connection: exceeded max-connections 800
In rare case, analytics service is crashed due to oom caused by geode service.
Environment
Cause
The problem itself is related to the Gemfire product used in the vROps. Gemfire is not handling clients' connections smoothly if the network link between the server and client is not stable enough.
In short, this was a problem in Gemfire which occurs in case of an unstable network.
Resolution
The problem has been addressed in the newer version of the Gemfire, but it is not consumed in the vROps yet.
Workaround:
Restarting vROPS cluster will resolve this issue shortly.
Replacing all Remote collector to Cloud proxy. It can reduce the frequency of problems.
Cloud proxy does not use gemfire anymore.
Additional Information
Remote collector / Cloud proxies are down status so there are collecting issue.
Feedback
