vSAN On-Disk format version upgrade fails with generic error "General vSAN error. Disk format conversion failed due to unexpected error"

Products

• VMware vSAN

Issue/Introduction

• Skyline health reports "Disk format version" alarm under Cluster > Monitor > vSAN > Skyline health.

• vSAN on-disk format upgrade may fail with below error.
  
  General vSAN error.
Disk format conversion failed due to unexpected error.

• On-disk format upgrade pre checks completes without any issues.
  
  

• var/log/vmware/vsan-health/vsan-health-service.log in vCenter report the Disk conversion thread met unexpected error.
  
  

  2025-05-14T06:23:52.737Z ERROR vsan-mgmt[1402576] [VsanVcDiskFormatConverterImpl::_Run opID=agw-0115827-5ae9] Disk conversion thread met unexpected error

  Traceback (most recent call last):

    File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4201, in _Run

    File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4956, in ConvertDisks

    File "bora/vsan/diskmgmt/vpxd/pyMoVsan/VsanVcDiskFormatConverterImpl.py", line 4385, in _RemediateBeforeConversion

    File "bora/vsan/clustermgmt/vpxd/VsanClusterPrototypeImpl.py", line 1252, in RemediateVsanCluster

    File "bora/vsan/clustermgmt/vpxd/VsanClusterPrototypeImpl.py", line 1268, in _RemediateVsanCluster

  2025-05-14T06:23:52.738Z INFO vsan-mgmt[1402576] [VsanVcDiskFormatConverterImpl::_PostConversion opID=agw-0115827-5ae9] Finished dfc for domain-c1065634 result:(vim.cluster.VsanUpgradeStatusEx) {

  } fault: (vim.fault.VsanFault) {

    faultMessage = (vmodl.LocalizableMessage) [

      (vmodl.LocalizableMessage) {

        key = 'com.vmware.vsan.diskconversion.msg.internalerror'

Environment

VMware vSAN (All Versions)

Cause

On-disk format version upgrade fails due to the incorrect permission on vCenter "vSAN health" folder and files.

Owner and Group for the folder /etc/vmware-vsan-health and the files inside it are incorrect, as listed below

root@esxi-011 [ /etc/vmware-vsan-health ]# ls -lth
total 34M
-rwxrwxrwx 1 root root 6.2K Jun  5 05:06 vsanhealthstatuschange.json
-rwxrwxrwx 1 root root  12K Jun  5 04:42 vsan-releases.json
-rwxrwxrwx 1 root root  15M Jun  3 19:42 vsanvcg.db
drwxr--r-x 3 root root 4.0K Nov  7  2024 licenses
-rwxrwxrwx 1 root root 3.0K Nov  7  2024 logger.conf
-rwxr--r-x 1 root root  19M Nov  7  2024 vsanvcg.db.bak
-rwxrwxrwx 1 root root 339K Nov  7  2024 vsanhealthremediation.yaml
-rwxrwxrwx 1 root root 4.7K Nov  7  2024 vsanhosthealthmanifest.json
-rwxrwxrwx 1 root root 2.5K Nov  7  2024 cloudhealthmanifest.json
-rwxrwxrwx 1 root root 8.2K Nov  7  2024 vsanhealthcorrelation.json
-rwxrwxrwx 1 root root 4.3K Nov  7  2024 vsanhealth.properties
drwxr--r-- 3 root root 4.0K Nov  7  2024 cloudHealthResources
-rwxrwxrwx 1 root root  726 Nov  7  2024 config.conf
-rwxr--r-x 1 root root  573 Nov  7  2024 silent.json

 

Resolution

vSAN health folder owner on vCenter needs to changed to "vsan-health".
Owner and group for some files need to be changed to "vsan-health" and "users".


Permission can be granted as below:

vSAN folder level permission where group is changed to "vmware-vsan-health"

root@VC#[/etc]# chown -R vsan-health:root vmware-vsan-health

 

File level permission can be granted as below where the users are changed to "vsan-health:users"

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users licenses/

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users logger.conf

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users vsanhealthstatuschange.json

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users vsan-releases.json

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users config.conf

root@VC#[/etc/vmware-vsan-health]# chown -R vsan-health:users silent.json

 

Permissions should look as below for a successful cluster remediation task to be passed.

 

root@VC#[ /etc/vmware-vsan-health ]# ls -lrth

total 34M

-rwxr--r-x 1 vsan-health users  573 Nov  7  2024 silent.json

-rwxrwxrwx 1 vsan-health users  726 Nov  7  2024 config.conf

drwxr--r-- 3 vsan-health root  4.0K Nov  7  2024 cloudHealthResources

-rwxrwxrwx 1 vsan-health root  4.3K Nov  7  2024 vsanhealth.properties

-rwxrwxrwx 1 vsan-health root  8.2K Nov  7  2024 vsanhealthcorrelation.json

-rwxrwxrwx 1 vsan-health root  2.5K Nov  7  2024 cloudhealthmanifest.json

-rwxrwxrwx 1 vsan-health root  4.7K Nov  7  2024 vsanhosthealthmanifest.json

-rwxrwxrwx 1 vsan-health root  339K Nov  7  2024 vsanhealthremediation.yaml

-rwxr--r-x 1 vsan-health root   19M Nov  7  2024 vsanvcg.db.bak

-rwxrwxrwx 1 vsan-health users 3.0K Nov  7  2024 logger.conf

drwxr--r-x 3 vsan-health users 4.0K Nov  7  2024 licenses

-rwxrwxrwx 1 vsan-health root   15M Jun  3 19:42 vsanvcg.db

-rwxrwxrwx 1 vsan-health users  12K Jun  5 04:42 vsan-releases.json

-rwxrwxrwx 1 vsan-health users 6.2K Jun  5 05:06 vsanhealthstatuschange.json

Additional Information

About the vSAN Disk Format

Understanding vSAN on-disk format versions and compatibility