Error: "Unable to establish a connection with Real-time metrics component. Please verify the Cloud Proxy status and review the logs."

Products

• VCF Operations

Issue/Introduction

• The "Real-time metrics" component (VODAP) has been deployed from the VCF Operations UI, Build > Lifecycle > Components as per Deploy Real-Time Metrics
• The Real-time Metrics Store (Build > Lifecycle > Components) is deployed with a Medium or Large size.
  (This can be verified by clicking on the Real-time Metrics Store link at the path mentioned.)
• An error message similar to the following may be seen in the VODAPAdapter log, located in directory: /storage/log/vcops/log/adapters/VODAPAdapter on the Unified Cloud Proxy
  
  Caused by: com.broadcom.ops.data.api.client.ApiException: Message:
HTTP response code: 500
HTTP response body:
{"status":"error","data":null,"pageInfo":null,"errorType":"INTERNAL_SERVER_ERROR","error":"Query API execution failed.","warnings":[],"infos":[]}


• In the VCF Operations UI, navigating to Operate > Workbench > Real Time Analysis > Overview, the following warning is displayed:
  
  Unable to establish a connection with Real-time metrics component. Please verify the Cloud Proxy status and review the logs.

Environment

VCF Operations 9.1.0

Resolution

1. Open VCF Operations UI and navigate to Build > Lifecycle > Components tab
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify a Control Plane node and take note of the IP address 
5. SSH to this Control Plane node as vmware-system-user 
6. Run the following commands providing password as necessary:
   
   sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf
   
   
7. Ensure that kubectl is working as expected and the SSH is connected to a Control Plane node by running the following command:
   
   kubectl get nodes -o wide
   
   NOTE: This kubectl command must complete sucessfully and return a list of nodes before continuing.
   
   
8. Download the script attached to this KB and save as repair-clickhouse-shard-ddl.sh
9. Using an SCP utility (e.g. Winscp), copy this downloaded file to /tmp directory on the same Control Plane node that the SSH session is connected to above.
10. Return to the SSH session and run the following commands:
    
    chmod +x /tmp/repair-clickhouse-shard-ddl.sh
bash /tmp/repair-clickhouse-shard-ddl.sh --dry-run
    
    
11. If the final line of the output from the previous command is as follows, DO NOT CONTINUE:
    
    All <N> ClickHouse pods report the same table count (<N>) in database 'vcf_ops_datastore'. No skew indicated by counts; nothing to repair. Exiting.
    
    
12. Only If the output of the previous command shows a difference in the count of Tables in the section:
    
    === Table counts per pod (read-only) ===
    
    Then run the following command:
    
    bash /tmp/repair-clickhouse-shard-ddl.sh --apply
    
    
13. Once the script has completed return to the VCF Operations UI and verify that Operate > Workbench > Real Time Analysis > Overview is now working as expected.