Metrics Streaming Failure alarm is seen in VMware NSX User Interface

Products

• VMware NSX

Issue/Introduction

Alarm Description:

Purpose: Indicates continuous streaming failure of metrics to VODAP (VCF Observability Data Platform) metrics collector

Impact: Dashboard in VODAP (VCF Observability Data Platform) metrics collector will not show the metrics from these nodes

Environment

VMware NSX 9.1

Resolution

Please confirm if VCF Observability Data Platform (VODAP) collector is deployed or not.

• If collector is deployed:
  • Get the status code for the streaming failure, included in the alarm description.
    
    
  • Potential issues and remediation based on status code:
    • UNAUTHENTICATED:
      • Check the root certificate that NSX API received:
        Login to the NSX Manager managing this node, get the global collector setting using API:
        GET https://<nsx-manager-ip>/policy/api/v1/infra/ops-global-config
        
        
      • Example output:
        {
    "operation_collectors": [
        {
            "root_certificate": "-----BEGIN CERTIFICATE-----\nXXXXXXXXXX\n-----END CERTIFICATE-----",
            "vidb_fqdn": "vibd-fqdn.example.com",
            "vidb_certificate": "YYYYYYY",
            "client_id": "vsp/########-####-####-####-########63e1/vsp-client",
            "client_secret": "########",
            "collector_type": "VODAP_STREAMING",
            "collector_fqdn": "collector-fqdn.example.com",
            "collector_port": 3050,
        }
    ],
}
        
        
      • Check the root certificate dispatched to ESXi/EDGE Transport Node:
        • ESXi:
          • SSH to the ESXi host.
          • Run the command below to get the output:
            /usr/lib/vmware/netopa/bin/sha-appctl -c get_collector_status --collector_type vodap -j
          • Example output:
            {
    "type": "vodap",
    "status": {
        "fqdn.example.com:3050": {
            "Global Config": "Received",
            "caching": "stopped",
            "properties": {
                "client": {
                    "VERSION": {
                        "major": 9,
                        "minor": 1,
                        "maintenance": 0,
                        "patch": 0,
                        "buildNumber": 88741123
                    }
                },
                "server": {
                    "VERSION": {
                        "major": 9,
                        "minor": 1,
                        "maintenance": 0
                    },
                    "COMPRESSION": false
                }
            },
        "state": "handshaked",
        "ingress_certificate": "-----BEGIN CERTIFICATE-----\nXXXXXXXXXX\n-----END CERTIFICATE-----"
}
 
        • Edge:
          • SSH to the Edge Transport Node as root.
          • Run the command below to get the output:
            /opt/vmware/nsx-netopa/bin/sha-appctl -c get_collector_status --collector_type vodap -j
            
            
      • If the output is "UNAVAILABLE" or "DEADLINE_EXCEED":
        • Login to the NSX Manager managing this node, get the global collector setting using API:
          GET https://<nsx-manager-ip>/policy/api/v1/infra/ops-global-config
          
          • Check if the "collector_fqdn" field in the "VODAP_STREAMING" collector in the API's response is same as {metrics_target_address} in the alarm description, and if the "collector_port" is same as {metrics_target_port}.
          • Check if DNS lookup result for "collector_fqdn" in the API's response is reachable or not.
          • Check and fix the connectivity from "Reported by node" to {metrics_target_address}:{metrics_target_port} via command:
            nc -n -v  {metrics_target_address}:{metrics_target_port}
          • Check if there is an extra firewall between TN node (including EDGE node and ESXi node) and VODAP collector, please check if there is a policy denying the tcp traffic from TN to the collector {metrics_target_address}:{metrics_target_port}.
            
            
• If VODAP collector is undeployed:
  • Run the APIs below directly on the NSX Manager managing this node:
    PATCH https://<nsx-manager-ip>/policy/api/v1/infra/ops-global-config with existing other type of collectors only to remove the VODAP_STREAMING type of collector.
  • Do a GET calling the same API to check if the VODAP_STREAMING type of collector is still present:
    GET https://<nsx-manager-ip>/policy/api/v1/infra/ops-global-config
  • If above remediation does not resolve the alarm, restart SHA agent on "Reported by node":
    • For NSX Edge:
      service nsx-sha restart
    • For ESXi host:
      /etc/init.d/netopad restart
                      

Maintenance window required for remediation? No