No metrics are available for the selected resource when accessing UI or querying via API

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

Users encounter "No metrics are available for the selected hosts and cluster" error when accessing the System→Overview page in SSP UI, specifically when refreshing flow processing capacity metrics.

This issue is not limited to host and can be seen for other resources as well.

Steps to Identify

Check metrics-query-server logs and look for google.protobuf.message.DecodeError entries and Error parsing message with type errors.

Steps:

1. If facing issue on UI: Login to SSP and goto affected page. Else if you are facing this issue with API: Call API again. This will trigger error condition in metrics-query-server

2. Login to SSPI

3. k get pod -A | grep metrics-query-server

4. k logs -n nsxi-platform <metrics-query-server pod name here> | grep 'google.protobuf.message.DecodeError'

If 'google.protobuf.message.DecodeError' log is present then SSP instance is impacted by this issue.

Environment

vDefend SSP >= 5.0

Cause

A race condition exists in the service responsible for persisting metrics. During Database reconnection windows, this causes corrupted blob data to be stored in the metrics database.

Resolution

This issue has been fixed in v5.1.1.1

Immediate Workaround

For SSP <= 5.1.1

Deploy the SQL recovery script to fix existing corruption and prevent future issues:

Step 1: Login to SSPI and Copy Recovery Script to PostgreSQL Pod

File: corrupt_blob_recovery_v4.sql

NAMESPACE = "nsxi-platform" POD = "metrics-postgresql-ha-postgresql-0" k cp corrupt_blob_recovery_v4.sql \ ${NAMESPACE} / ${POD} :/tmp/corrupt_blob_recovery_v4.sql

Step 2: Execute Recovery Script

k exec -it -n ${NAMESPACE} ${POD} -- bash PGPASSWORD = $POSTGRES_PASSWORD \ psql -w -U postgres -d metrics \ -f /tmp/corrupt_blob_recovery_v4.sql

Step 3: Restart metrics-query-server deployment

k rollout restart deploy metrics-query-server -n nsxi-platform

Post-Upgrade Requirement for SSP v5.1.1.1: Although the fix is included in SSP v5.1.1.1, it does not retroactively clean up existing database corruption. Because automatic recovery is not available until next release of SSP, you must run the one-shot recovery script once after upgrading to v5.1.1.1 to resolve any pre-existing issues.

For SSP >= 5.1.1.1

Run one-shot recovery script to fix existing corruption:

Step 1: Login to SSPI and Copy Recovery Script to PostgreSQL Pod

File: corrupt_blob_recovery_oneshot.sql

NAMESPACE = "nsxi-platform"

--- If SSP = 5.1.1.1
POD = "metrics-postgresql-ha-postgresql-0"

k cp corrupt_blob_recovery_oneshot.sql ${NAMESPACE}/${POD}:/tmp/corrupt_blob_recovery_oneshot.sql

Step 2: Execute Recovery Script

k exec -it -n ${NAMESPACE} ${POD} -- bash

--- If SSP = 5.1.1.1

PGPASSWORD=$POSTGRES_PASSWORD psql -w -U postgres -d metrics -f /tmp/corrupt_blob_recovery_oneshot.sql

Step 3: Restart metrics-query-server deployment

k rollout restart deploy metrics-query-server -n nsxi-platform

Internal ticket number for this issue is #3679596.