"Unable to query the data. Try again later" error on UI when clicking on vSAN performance tab to view performance

Products

• VMware vSAN

Issue/Introduction

Symptoms:

vSAN performance data may not show up and gives an error "Unable to query the data. Try again later" when clicking on any vSAN performance tabs.

vSAN health may show warning for vSAN performance service.

Under vSAN services tab, you may see vSAN performance service is enabled and healthy.

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

The performance stats collection and display may not work when the performance stats object (.vsan.stats) path is in Missing state.

This can happen when actual vSAN datastore UUID is vsan:526##########-5d1##########a9 and the .vsan.stats object path shows as vsan:529##########-5f5##########/.vsan.stats. which is not valid.

This would not let the vSAN to read or write the performance stats on the object and would cause the stats unavailability. 

This can be verified by listing the vSAN performance stats object UUID from one of the ESXi host which is part of vSAN cluster.

[root@esx] localcli vsan debug object list -u f8950c5f-e0ab-2a44-4a3d-########

Object UUID: f8950c5f-e0ab-2a44-4a3d-########:

---

Type: vmnamespace

Path: /vmfs/volumes/vsan:529f2ce9f6e02b45-########/.vsan.stats (Missing)
Group UUID: f8950c5f-e0ab-2a44-4a3d-########
Directory Name: .vsan.stats

Resolution

Recommended to fix the .vsan.stats path issue by disabling and re-enabling the vSAN performance service using the steps below.

- Identify and delete the current unhealthy .vsan.stats object.

• Login to the SSH session for one of the ESXi host which is part of vSAN cluster.

• Identify and verify the vSAN perf service object UUID.

• Delete the unhealthy perf stats object using command line.

 /usr/lib/vmware/osfs/bin/objtool delete -u f8950c5f-e0ab-2a44-4a3d-########

- Once the object is deleted, try to enable the vSAN performance service from the vCenter server RVC using the instructions below.

- Login to vCenter server SSH session and enter into bash mode using shell command.

• Run command 'rvc localhost'

• This would prompt for credentials for [email protected] user.

• enter the password.

Example:

# rvc localhost
[DEPRECATION] The trollop gem has been renamed to optimist and will no longer be supported. Please switch to optimist as soon as possible.
Install the "ffi" gem for better tab completion.
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.
Using default username "[email protected]".
password:

• Navigate to the cluster using cd localhost/Datacenter-name/computers/Cluster-name/ command.

• Verify the current status of the service and perf stats object.

localhost/Datacenter-name/computers/Cluster-name> vsan.perf.stats_object_info .
Directory Name: unknown
vSAN Object UUID: (Not found)
SPBM Profile: None
vSAN Policy:
localhost/Datacenter-name/computers/Cluster-name/

• Create a new .vsan.stats object using the command below.

localhost/Datacenter-name/computers/Cluster-name> vsan.perf.stats_object_create .
Creating vSAN Stats DB object, which will enable vSAN Performance Service ...
Task: Enable vSAN performance service
New progress: 1%
Task result: success
localhost/Datacenter-name/computers/Cluster-name>

• Verify the new perf stats object health using the command below.

localhost/Datacenter-name/computers/Cluster-name> vsan.perf.stats_object_info .
Directory Name: .vsan.stats
vSAN Object UUID: 637e6768-7ec3-7aa2-c510-########
SPBM Profile: vSAN Default Storage Policy

-------

  Extended attributes:
    Address space: 273804165120B (255.00 GB)
    Object class: vmnamespace
    Object path: /vmfs/volumes/vsan:526########-5d1########/.vsan.stats
    Object capabilities: NONE

localhost/Datacenter-name/computers/Cluster-name>

 

• Verify the vSAN performance tab and it should not be giving an error and the performance data would start showing up after five minutes which is default interval.

When .vsan.stats object which is also called as vSAN perf object is in unhealthy state, the stats collection and display would fail. Re-creating the .vsan.stats object is expected to fix the object health and stats display to work as expected.