Alert "Http Metric Requests Dropped" active for Supervisor object

Products

• VCF Operations

Issue/Introduction

• The VCF Operations dashboard Kubernetes VKS Clusters does not have any data from the Pods and Containers in the VKS Clusters
• The alert "Https Metric Requests Dropped" is active on the Supervisor objects in VCF Operations

Environment

VCF Operations 9.0.x

Cause

The supervisor-management-proxy-ca-*** secret in the VKS cluster has expired.

Resolution

To validate if your environment is affected by this issue:

1. Open a SSH session to the Supervisor Control Plane (See Troubleshooting vSphere Supervisor Control Plane VMs)

2. Find the name of the Supervisor Managment Proxy namespace

   kubectl get namespaces

   Find the name that begins with svc-supervisor-management-proxy-

3. Find the name of the Supervisor Management Proxy pod

   kubectl get pods -n <namespace_name_from_step_2>

   Find the name that begins with supervisor-management-proxy-

4. Review the logs from the Supervisor Management Proxy pod

   kubectl logs -n <namespace_name_from_step_2> <pod_name_from_step_3>

5. Look for log entries like below

   [YYYY-MM-DDThh:mm:ss.####] "- - -" 0 - 0 0 # - "-" "-" "-" "-" "-"

6. Connect to the VKS Cluster with the method you have previously configured (See Connect to a VKS Cluster with VCF CLI)

7. Find the namespace that was defined when telegraf was installed in the VKS cluster

   kubectl get namespaces

8. Find the name of the telegraf pod(s)

   kubectl get pods -n <namespace_name_from_step_7>

9. Review the logs from the telegraf pod(s)

   kubectl logs -n <namespace_name_from_step_7> <pod_name_from_step_8>

10. Look for log entries like below

    E! [agent] Error writing to outputs.http: Post "https://supervisor-management-proxy.default.svc.cluster.local:#####/arc/tkgs/metric": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time YYYY-MM-DDThh:mm:ssZ is after YYYY-MM-DDThh:mm:ssZ

 

If the log entries match the entries in these steps, Contact Broadcom Support to open a support case and reference this KB.