vDefend SSP Alarm: Metrics Service status is degraded or down

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Alarms with any of the following description - 

"Metrics service {service-name} is degraded"
"Metrics service {service-name} is down"

e.g 
Metrics service metrics-manager is degraded
Metrics service metrics-query-server is down

The {service-name} can be any of the following

1. 1. metrics-manager
   2. metrics-query-server
   3. metrics-app-server
   4. metrics-db-helper
   5. metrics-postgresql-ha-postgresql
   6. metrics-postgresql-ha-pgpool

If the alarm stays open for more than 30 minutes or if its occurring multiple times, please proceed to the Resolution section

Environment

vDefend SSP >= 5.0

Cause

One or more replica pods of Metrics service {service-name} are not in a running state

Resolution

• Maintenance window required for remediation?
  No

  Steps to resolve:

  Try re-starting the deployment/statefulset. This should take care of any transient issues

  • • Log into SSPI root shell
    • If {service-name} is metrics-postgresql-ha-postgresql

      • Run 'k rollout restart statefulset metrics-postgresql-ha-postgresql -n nsxi-platform'

    • else

      • Run 'k rollout restart deployment {service-name} -n nsxi-platform'

  Wait for ~5 minutes and  check if the {service-name} pods recover

  • • • 'k get pods -n nsxi-platform | grep {service-name}'

  Wait for ~20 minutes and check if the alarm is auto-resolved 

   

   If the alarm persists, check for the following 

  • Check for disk usage alarms,  https://knowledge.broadcom.com/external/article?articleNumber=384110
  • Check for memory usage alarms, https://knowledge.broadcom.com/external/article?articleNumber=384111
  • Check for CPU usage alarms, https://knowledge.broadcom.com/external/article?articleNumber=384109

  If none of the above is applicable, please open a ticket with Broadcom Support.