All disks on version but with older vSAN objects after upgrade

Products

• VMware vSAN

Issue/Introduction

Symptoms:

.vsan.stats namespace object may not upgrade from version 7 to 10 when upgrading to 6.7U3 or from version 13 to 14 when upgrading to version 7.0U2

A vCenter alert shows that a vSAN object is on an older version

From the command line of an ESXi node you see the following:

esxcli vsan debug object overview
Object UUID                           Group UUID                            Version  Size       Used     SPBM Profile                 Healthy Components
------------------------------------  ------------------------------------  -------  ---------  -------  ---------------------------  ------------------
########-####-####-####-########6fbb  ########-####-####-####-########6fbb       10  40.00 GB   0.01 GB  vSAN Default Storage Policy  3 of 3
########-####-####-####-########6fbb  ########-####-####-####-########6fbb       10  255.00 GB  0.68 GB  vSAN Default Storage Policy  3 of 3
########-####-####-####-########6fc3  ########-####-####-####-########6fc3       7  255.00 GB  1.36 GB  vSAN Default Storage Policy  3 of 3

 esxcli vsan debug object list|grep -E 'Object UUID:|Version:|Path:'|grep -ve objectVersion:

Object UUID: ########-####-####-####-########6fbb
   Version: 10
   Path: /vmfs/volumes/vsan:524cb6a6########-######4149fc60df/########-####-####-####-########6fbb/.VM1.vmdk (Exists)
Object UUID: ########-####-####-####-########6fbb
   Version: 10
   Path: /vmfs/volumes/vsan:524cb6a6########-######4149fc60df/.VM1 (Exists)
Object UUID: ########-####-####-####-########6fc3
   Version: 7
   Path: /vmfs/volumes/vsan:524cb6a6########-######4149fc60df/.vsan.stats (Exists)

Note: The preceding command excerpts are only examples. Environmental variables may vary depending on the customer's environment.

Environment

VMware vSAN 6.7.x
VMware vSAN 7.0.x

VMware vSAN 8.0.x

Resolution

Option 1:

1) Change the storage policy by doing the following:
   a) Click on the cluster name in the left hand pane
   b) select Configure>vSAN Services
   c) Click on Edit to the left of Performance Service
   d) Select a different storage policy from the drop down list
   e) From RVC drilled down to the cluster level run vsan.ondisk_upgrade to start a manual ondisk upgrade
2) Run esxcli vsan debug object overview to confirm all objects are on version 10
3) If successful and the original Storage policy for Performance Service is desired, set it back

Note: Changing the storage policy can work for vmdks/namespaces as well that may have failed to upgrade to version 10

Option 2: 
If option 1 doesn't resolve the issue try the following:
Disable Performance Service and then re-enable, this will create a new .vsan.stats namespace with version 10

To disable vSAN Performance

1) Select the vSAN cluster
2) Select Configure Tab
3) Under vSAN select Services
4) Select Performance Service and click Edit
5) Click the toggle switch to disable performance service and click Apply

To enable vSAN Performance
1) Since you should already still be at vSAN Services
2) Click Edit to the right of Performance Service
3) Click the toggle switch to enable performance service and click Apply

Note: 

• This will delete any perf stats already collected. Make sure to mention this to the customer in case they are in the middle of troubleshooting a performance issue and need to save the data.
• There is no Disable option in vSAN 8/9 vSAN Performance Service Setting in vCenter UI, for version 8 and above follow KB Unable to Disable Performance Service in vSAN 8/9