vSAN -- vSAN Health Service - Performance Service - Stats DB object conflicts

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

Introduction: 

Q. What does the Performance Service - Stats DB object conflicts check do?

A. The check verifies whether there are multiple Stats DB objects present on the vSAN datastore.

Q. What does it mean when it is in an error state?

A. Multiple Stats DB objects were found on the vSAN Datastore of the associated vSAN Cluster.

The related directories were renamed by appending a number to the directory name (e.g. "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)" etc. )

This is an issue because per vSAN Datastore there can be only one Stats DB Object (= Directory named ".vsan.stats" without any number appended). 

 

Symptoms: 

Warning in vSAN Health:

 

The Error details (when clicked on "Troubleshooting"), show one or more directories named "vsan.stats" with a number appended

Naming Examples: "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)"

 

Alert Examples from different Clusters: 

 

Example A.): 

 

Example B.): 

 

Via Datastore Browser: Multiple Folders named "vsan.stats" are shown

 

Examples from different Clusters: 

 

Example A.): 

 

Example B.): 

 

 

 

Via SSH/Putty to any of the vSAN Hosts: Multiple Directories named "vsan.stats" are shown

 

Example: 

 

 

 

 

You might observe: When trying to access one of the folders you get the error: "Operation failed! File [vsanDatastore] ########-####-####-######### was not found" 

 

Example:

 

 

 

Environment

vSAN (All Versions)

Cause

This issue occurs when multiple Stats DB objects are present on the vSAN datastore. vSAN requires exactly one Stats DB Object (the directory named .vsan.stats) per datastore. Additional directories are typically created during cluster maintenance, hardware failures (absent disks), or incomplete upgrades.

Resolution

All Folders/Directories containing a number in their "vsan.stats" name need to be deleted.

Examples: "vsan.stats-1", "vsan.stats-2", "vsan.stats-3", "vsan.stats(1)", "vsan.stats(2)" etc.

 

 

Steps:

 

1.)     vsan.stats folders contain historical vSAN performance data. If this data is required for audit purposes or would like to be kept, make a Backup of these folders by using the Datastore Browser

Note: Merging of multiple Stats DB objects is not supported.

( If needed, see here for more info in regards to vsan.stats reflected by vSAN Performance Service being enabled )

 

 

2.)    Via Datastore Browser: Delete the "vsan.stats" folders/directories which contain a number in their name (as outlined above)

 

 

 

 

3.)    In vSAN Healthcheck: Verify that the Warning "Stats DB object conflicts" is now gone by hitting retest in vSAN Health

 

 

If assistance is required, please open a Ticket with VMware by Broadcom Support.

Additional Information

vSAN Health Service - Performance Service - Status Check

vSAN Health Service - Performance Service - Stats DB object check