vDefend / Avi Alarm : Metrics service {service-name} is degraded OR down

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

You are running SSP, License Hub for vDefend and Avi, or Avi Operations and have encountered an alarm with the description

"Metrics service {service-name} is degraded"
"Metrics service {service-name} is down"

e.g 
Metrics service metrics-manager is degraded
Metrics service metrics-query-server is down

The {service-name} can be any of the following

1. 1. metrics-manager
   2. metrics-query-server
   3. metrics-app-server
   4. metrics-db-helper
   5. metrics-postgresql-ha-postgresql
   6. metrics-postgresql-ha-pgpool

If the alarm stays open for more than 30 minutes or if its occurring multiple times, please proceed to the Resolution section

Environment

vDefend SSP Version: 5.0 and later
License Hub for vDefend and Avi: 2.0 and later
Avi Operations: 5.2 and later

Cause

One or more replica pods of Metrics service {service-name} are not in a running state

Resolution

Maintenance window required for remediation?
No

Steps to resolve:

Try re-starting the deployment/statefulset. This should take care of any transient issues

Note: If multiple SSP instances exist within SSPI, ensure you export the target SSP instance context before executing any kubectl / k

• • Log into the Appliance VM shell
  • If {service-name} is metrics-postgresql-ha-postgresql

    • Run 'k rollout restart statefulset metrics-postgresql-ha-postgresql -n nsxi-platform'

  • else

    • Run 'k rollout restart deployment {service-name} -n nsxi-platform'

Wait for ~5 minutes and  check if the {service-name} pods recover

• • • 'k get pods -n nsxi-platform | grep {service-name}'

Wait for ~20 minutes and check if the alarm is auto-resolved 

 If the alarm persists, check for the following 

• Check for disk usage alarms,  https://knowledge.broadcom.com/external/article?articleNumber=384110
• Check for memory usage alarms, https://knowledge.broadcom.com/external/article?articleNumber=384111
• Check for CPU usage alarms, https://knowledge.broadcom.com/external/article?articleNumber=384109

If none of the above is applicable, please open a ticket with Broadcom Support.