vSAN -- vSAN Health Service - Performance Service

Products

VMware vSAN VMware vSAN 7.x VMware vSAN 8.x

Issue/Introduction

Introduction:

Q. What does the Performance Service - Stats DB object conflicts check do?

A. The check verifies whether there are multiple Stats DB objects present on the vSAN datastore.

Q. What does it mean when it is in an error state?

A. Multiple Stats DB objects were found on the vSAN Datastore of the associated vSAN Cluster.

The related directories were renamed by appending a number to the directory name (e.g. "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)" etc. )

This is an issue because per vSAN Datastore there can be only one Stats DB Object (= Directory named ".vsan.stats" without any number appended).

Symptoms:

Warning in vSAN Health:

The Error details (when clicked on "Troubleshooting"), show one or more directories named "vsan.stats" with a number appended

Naming Examples: "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)"

Alert Examples from different Clusters:

Example A.):

Example B.):

Via Datastore Browser: Multiple Folders named "vsan.stats" are shown

Examples from different Clusters:

Example A.):

Example B.):

Via SSH/Putty to any of the vSAN Hosts: Multiple Directories named "vsan.stats" are shown

Example:

You might observe: When trying to access one of the folders you get the error: "Operation failed! File [vsanDatastore] ########-####-####-######### was not found"

Example:

Environment

vSAN (All Versions)

Cause

The issue can come up after a Cluster merge or Maintenance activities.

Resolution

All Folders/Directories containing a number in their "vsan.stats" name need to be deleted.

Examples: "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)" etc.

Steps:

1.) If unsure whether the data these vsan.stats folders might contain would be relevant to your environment for further purpose, make a Backup of these folders by using the Datastore Browser

Note: Merging of multiple Stats DB objects is not supported.

( If needed, see here for more info in regards to vsan.stats reflected by vSAN Performance Service being enabled )

2.) Via Datastore Browser: Delete the "vsan.stats" folders/directories which contain a number in their name (as outlined above)

3.) If there were issues with executing Step 2.), use command line (via logging into any of the vSAN Hosts via SSH/Putty)

3.1) Get the Group UUIDs associated with Folders/Directories containing a number in their "vsan.stats" name

esxcli vsan debug object list |grep -B1 "Directory Name: .vsan.stats"

Example:

3.2) Based on Step 3.1) --> Delete the Group UUIDs associated with Directories named ".vsan.stats-NUMBER" (e.g. as shown above: .vsan.stats-3)

/usr/lib/vmware/osfs/bin/objtool delete -u <GROUP UUID> -f

Note: Once the command is executed, the deleted Group UUID cannot be recovered (= it is gone)

Example below, shows that we are deleting the Group UUID associated with Directory ".vsan.stats-3")

4.) Restart the vSAN Management Service on all vSAN Hosts in the Cluster by logging into the Hosts via SSH/Putty:

/etc/init.d/vsanmgmtd restart

/etc/init.d/vsanmgmtd status

5.) In vSAN Healthcheck: Verify that the Warning "Stats DB object conflicts" is now gone by hitting retest in vSAN Health

If assistance is required, please open a Ticket with VMware by Broadcom Support.

Additional Information

vSAN Health Service - Performance Service - Status Check

vSAN Health Service - Performance Service - Stats DB object check