vCenter Management Interface (VAMI) displays "0% of Undefined" disk utilization and empty CPU metrics

Products

VMware vCenter Server

Issue/Introduction

The vCenter Appliance Management Interface (VAMI) under the Monitor tab displays "0% of Undefined" for disk utilization and CPU metrics also appear empty for every timeframe:
The /var/log/vmware/observability/observability.log file on vCenter Server contains errors such as
[YYYY-MM-DDTHH:MM:SS]ERROR615086[Thread-46] - com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:438) - Not a JSON Object: "MB"
java.lang.IllegalStateException: Not a JSON Object: "MB"
at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:91)
at com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:384)
at com.vmware.observability.observer.SqlWriter.execute(SqlWriter.java:298)
at com.vmware.stellar.plugin.Daemon.drain(Daemon.java:183)
at com.vmware.stellar.executor.DaemonExecutor.run(DaemonExecutor.java:143)
at java.lang.Thread.run(Thread.java:750)
[YYYY-MM-DDTHH:MM:SS]ERROR615087[Thread-46] - com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:438) - Not a JSON Object: "filesystem"
java.lang.IllegalStateException: Not a JSON Object: "filesystem"
at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:91)
at com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:384)
at com.vmware.observability.observer.SqlWriter.execute(SqlWriter.java:298)
at com.vmware.stellar.plugin.Daemon.drain(Daemon.java:183)
at com.vmware.stellar.executor.DaemonExecutor.run(DaemonExecutor.java:143)
at java.lang.Thread.run(Thread.java:750)
[YYYY-MM-DDTHH:MM:SS]ERROR615087[Thread-46] - com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:438) - Not a JSON Object: "1776101272594"
java.lang.IllegalStateException: Not a JSON Object: "1776101272594"
at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:91)
at com.vmware.observability.observer.SqlWriter.writeMetricInfoToStatDB(SqlWriter.java:384)
at com.vmware.observability.observer.SqlWriter.execute(SqlWriter.java:298)
at com.vmware.stellar.plugin.Daemon.drain(Daemon.java:183)
at com.vmware.stellar.executor.DaemonExecutor.run(DaemonExecutor.java:143)
at java.lang.Thread.run(Thread.java:750)
Within the '/storage/db/vmware-observability/stellar.db' database the metric to collect the data is disabled. Example:
- Open an SSH session to the vCenter, login as root and run the following commands:
  root@<vCenter>[ / ]# sqlite3 /storage/db/vmware-observability/stellar.db
  SQLite version 3.38.5 <Timestamp>
  Enter ".help" for usage hints.
  sqlite> SELECT name, status FROM metrics WHERE source = "stat";
  user|0
  nice|0
  system|0
  idle|0
  iowait|0
  irq|0
  softirq|0
  steal|0
  guest|0
  guest_nice|0
  cpuUsage|0
  sqlite> SELECT name, status FROM metrics WHERE source = "filesystem";
  totalSpace|0
  freeSpace|0
  usedSpace|0
  readOnly|0
  type|0
  diskName|0
  sqlite>
- Notice that all the values above are set to 0.

Environment

VMware vCenter Server

Cause

The issue is caused by a failure in the VMware Observability Service. This service is responsible for collecting and pushing partition and CPU metrics to the management layer. The service fails to report data because the collection flags for the stat and filesystem metric sources are disabled (set to status = 0) within the internal stellar.db configuration database.

Resolution

To resolve this issue, the metric status flags must be manually enabled in the observability service database.

NOTE: Take a Snapshot/Backup of the vCenter Server before making any changes to the database.

Open an SSH session to the vCenter Server and login as root user.
Access the observability database using the sqlite3 utility:
```
sqlite3 /storage/db/vmware-observability/stellar.db
```
Verify the current status of the metrics as per the step provided above. If the values return 0, they are disabled and need to be enabled again:
```
UPDATE metrics SET status = 1 WHERE source IN ('stat', 'filesystem');
```

Confirm the update by running the SELECT query again, all relevant metrics should now show a status of 1:

sqlite> SELECT name, status FROM metrics WHERE source = "stat";
user|1
nice|1
system|1
idle|1
iowait|1
irq|1
softirq|1
steal|1
guest|1
guest_nice|1
cpuUsage|1
sqlite> SELECT name, status FROM metrics WHERE source = "filesystem";
totalSpace|1
freeSpace|1
usedSpace|1
readOnly|1
type|1
diskName|1

Restart the VMware Observability Service to apply the changes:
```
service-control --restart observability
```
Wait 5–10 minutes for the metrics to populate. Refresh the VAMI page to verify that Disk and CPU utilization are now correctly displayed.