vRealize Operations Manager does not Collect data
Article ID: 325007
Updated On:
Products
VMware Aria Suite
Issue/Introduction
Symptoms:
- Timeout error messages sporadically appearing in the VMware vROps User Interface
- Adapters status is “No Data Receiving”
- In the /usr/lib/vmware-vcops/user/log/gemfire_vRealize Ops Analytics_*.log, you see entries similar to:
[severe 2017/06/18 00:19:11.158 UTC <locator request thread[2349]> tid=0x176f] SSL Error in connecting to peer /X.X.X.X[35,509].
javax.net.ssl.SSLException: Received close_notify during handshake
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1901)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2002)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at com.gemstone.gemfire.internal.SocketCreator.configureServerSSLSocket(SocketCreator.java:1077)
at com.gemstone.org.jgroups.stack.tcpserver.TcpServer$3.run(TcpServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
javax.net.ssl.SSLException: Received close_notify during handshake
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1901)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2002)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at com.gemstone.gemfire.internal.SocketCreator.configureServerSSLSocket(SocketCreator.java:1077)
at com.gemstone.org.jgroups.stack.tcpserver.TcpServer$3.run(TcpServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
- In the /usr/lib/vmware-vcops/user/log/gemfire_vRealize Ops Collector_*.log, you see entries similar to:
[warning 2017/06/18 00:10:35.370 UTC vRealize Ops Collector-05f930-fac-49-8bb7-1d97 <poolLoadConditioningMonitor-CollectorAnalyticsPool> tid=0x4d] Could not connect to: X.X.X.X:10000
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at com.gemstone.gemfire.internal.SocketCreator.configureClientSSLSocket(SocketCreator.java:1141)
at com.gemstone.gemfire.internal.SocketCreator.connect(SocketCreator.java:987)
at com.gemstone.gemfire.internal.SocketCreator.connect(SocketCreator.java:945)
at com.gemstone.gemfire.internal.SocketCreator.connectForClient(SocketCreator.java:914)
at com.gemstone.gemfire.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:97)
at com.gemstone.gemfire.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:134)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl.createLifetimeReplacementConnection(ConnectionManagerImpl.java:991)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl$ConnectionMap.checkLifetimes(ConnectionManagerImpl.java:1524)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl$LifetimeExpireConnectionsTask.run(ConnectionManagerImpl.java:842)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at com.gemstone.gemfire.internal.SocketCreator.configureClientSSLSocket(SocketCreator.java:1141)
at com.gemstone.gemfire.internal.SocketCreator.connect(SocketCreator.java:987)
at com.gemstone.gemfire.internal.SocketCreator.connect(SocketCreator.java:945)
at com.gemstone.gemfire.internal.SocketCreator.connectForClient(SocketCreator.java:914)
at com.gemstone.gemfire.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:97)
at com.gemstone.gemfire.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:134)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl.createLifetimeReplacementConnection(ConnectionManagerImpl.java:991)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl$ConnectionMap.checkLifetimes(ConnectionManagerImpl.java:1524)
at com.gemstone.gemfire.cache.client.internal.pooling.ConnectionManagerImpl$LifetimeExpireConnectionsTask.run(ConnectionManagerImpl.java:842)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
- Repeated exception error for com.gemstone.gemfire.cache.client.NoAvailableServersException
- Network instability in terms of number of dropped packets in the analytics nodes communication
- SYN_RECV sockets existence on the vROps nodes
Environment
VMware vRealize Operations Manager 6.1.x
VMware vRealize Operations Manager 6.3.x
VMware vRealize Operations Manager 6.2.x
VMware vRealize Operations Manager 6.4.x
VMware vRealize Operations Manager 6.6.x
VMware vRealize Operations Manager 6.0.x
VMware vRealize Operations Manager 6.5.x
VMware vRealize Operations Manager 6.3.x
VMware vRealize Operations Manager 6.2.x
VMware vRealize Operations Manager 6.4.x
VMware vRealize Operations Manager 6.6.x
VMware vRealize Operations Manager 6.0.x
VMware vRealize Operations Manager 6.5.x
Cause
This is occurs by 3rd party Gemfire/GEODE component not handling correctly its internal SSL Socket creating an operation flow.
Resolution
This is a known issue affecting vRealize Operations Manager 6.6 and earlier versions.
The issue is fixed in vRealize Operation Manager 6.6.1, To download, Download VMware vRealize Operations Manager 6.6.1
In vRealize Operations Manager 6.6.1 the resiliency to networking issues has been improved and few flows suffering of the same problem have been covered and resolved.
Workaround:
If you are unable to upgrade to vRealize Operations Manager 6.6.1, perform the workaround by restarting vROps cluster as below.
From, vROps Admin UI > Take the vROPS cluster Offline and then bring Online.
Note: Ensure to verify and resolve if there are any network issues.
The issue is fixed in vRealize Operation Manager 6.6.1, To download, Download VMware vRealize Operations Manager 6.6.1
In vRealize Operations Manager 6.6.1 the resiliency to networking issues has been improved and few flows suffering of the same problem have been covered and resolved.
Workaround:
If you are unable to upgrade to vRealize Operations Manager 6.6.1, perform the workaround by restarting vROps cluster as below.
From, vROps Admin UI > Take the vROPS cluster Offline and then bring Online.
Note: Ensure to verify and resolve if there are any network issues.
Feedback
Yes
No
Powered by
