Disk Format Converter (DFC) pre-check fails for a HCI Mesh / Standard vSAN environment with "Detected host(s) are running in mixed mode, please upgrade ESX servers to the same version."

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN HCI Mesh environment
  
  
• Remote vSAN datastore
  
  
• You see the below messages in the vsan health service logs

2021-01-29T06:43:01.804Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_CheckCluster opID=noOpId] 'vim.HostSystem:host-80': targetVersion 14
2021-01-29T06:43:01.977Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-104': targetVersion 14
2021-01-29T06:43:01.978Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-117': targetVersion 14
2021-01-29T06:43:01.978Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-121': targetVersion 14
2021-01-29T06:43:01.979Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-129': targetVersion 13
2021-01-29T06:43:01.980Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-133': targetVersion 13
2021-01-29T06:43:01.980Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_ComputeTargetVersionForCluster opID=noOpId] 'vim.HostSystem:host-137': targetVersion 13
2021-01-29T06:43:02.118Z INFO vsan-mgmt[64234] [VsanClusterConfigHelper::GetRemoteVersionsSupported opID=noOpId] Supported version list for client cluster version 14: [13, 14]
2021-01-29T06:43:04.569Z INFO vsan-mgmt[64234] [VsanVcDiskFormatConverterImpl::_RunPreCheck opID=noOpId] Finished precheck on 'vim.ClusterComputeResource:domain-c164': (vim.cluster.VsanDiskFormatConversionCheckResult) {
issues = (vim.VsanUpgradeSystem.PreflightCheckIssue) [
(vim.vsan.upgradesystem.MixedEsxVersionIssue) {
msg = 'Detected host(s) are running in mixed mode, please upgrade ESX servers to the same version.'
}
],
isSupported = false,
isDataMovementRequired = false
}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

• This issue is also observed in the standard vSAN Cluster, where vSAN on disk format precheck upgrade fails with the following error

  "Detected host(s) are running in mixed mode, please upgrade ESX servers to the same version."

  

 

Environment

VMware vSAN 7.0.x

VMware vSAN 8.0.x

Cause

vSAN on-disk format upgrade will only work when all the vSAN nodes are on the same ESXi version. If there are mixed versions, then the vSAN on-disk format upgrade will fail. 

 

The client & server clusters participating in the HCI Mesh are on different ESXi versions, resulting in the DFC pre-check failing with "Detected host(s) are running in mixed mode, please upgrade ESX servers to the same version."

HCI Mesh checks the server cluster's compatibility with the client cluster as part of the DFC pre-check.

 

 

Resolution

vSAN On disk format Pre-check Upgrade analyzes the cluster to uncover any issues that might prevent a successful upgrade.

• For vSAN HCI Mesh, ensure all vSAN clusters participating in the HCI Mesh are on the same ESXi version.
  
  
• For vSAN Standard Cluster: Ensure all vSAN cluster nodes are running the same ESXi version.
  
  Note :
  
  
  • Upgrade any ESXi hosts that are on older versions so that all nodes in the cluster are consistent.
  • Once all nodes are on the same ESXi version, re-run the vSAN on-disk format upgrade precheck.
  • If the vSAN precheck is successful the perform a vSAN on disk format upgrade by following the guidelines mentioned in Broadcom Techdocs Upgrading vSAN Disk Format Using vSphere Client