" 500 Internal Server Error" after logging into Aria Operations

Products

• VCF Operations

Issue/Introduction

- Everytime we login to vROPS, 500 Internal Server Error pops up as below.

- One of the nodes in cluster still stuck in "Analytics is Starting"

- Integration Objects are not being collected, most of the Adapters shows "Warning" under Integrations, and Gemfire Fails to execute "Analytics.getMetrices"

Environment

Aria Operations 8.18.x

Cause

Gemfire service fails with analytics logs showing "Analytics not registered"

YYYY-MM-DDTHH:MM:SS,957+0000 ERROR [ServerConnection on port 10000 Thread 655] [HM9qYLPdE2ewhPb3fIKKk3j6o1AynC4k] com.vmware.vcops.controller.data.utils.ResourceAccessorBase.getAnalyticsResourceInfo - There was error while retrieving state and status: AnalyticsInterface.getResourcesStateStatus failed : FunctionException: org.apache.geode.cache.execute.FunctionException: com.vmware.vcops.platform.gemfire.exceptions.FunctionNotRegisteredException: AnalyticsInterface.getResourcesStateStatus is not registered.
YYYY-MM-DDTHH:MM:SS,969+0000 ERROR [ServerConnection on port 10000 Thread 655] [HM9qYLPdE2ewhPb3fIKKk3j6o1AynC4k] com.vmware.vcops.federation.BasicExecutorInvoker.execute - Failed to execute function 'AnalyticsInterface.getMetricValues : FunctionException: org.apache.geode.cache.execute.FunctionException: com.vmware.vcops.platform.gemfire.exceptions.FunctionNotRegisteredException: AnalyticsInterface.getMetricValues is not registered.org.apache.geode.cache.execute.FunctionException: org.apache.geode.cache.execute.FunctionException: com.vmware.vcops.platform.gemfire.exceptions.FunctionNotRegisteredException: AnalyticsInterface.getMetricValues is not registered.

Resolution

Step1:

Check validity of the Certificates of the endpoints, KB 315906 (How to renew an adapter certificate or clear the Expired Certificates banner in Aria Operations) 

Step2:

Check if there are any storage issues on the vROPS, KB 318408 (Troubleshooting Storage Issues in Aria Operations) 

Step3:

1. Take the cluster offline in sequence as in the KB,  KB 341964 (Shutdown and Startup sequence for VCF/Aria Operations cluster) 

2. Wait for 5 min and Bring the cluster back Online