Upgrading vSAN On-disk format version fails and remains at v3
search cancel

Upgrading vSAN On-disk format version fails and remains at v3

book

Article ID: 326420

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:
  • vSAN object versions still at v3 after On-disk Format v5 upgrade
  • These issues are seen when the vCenter Server version is at 6.7 while hosts are at ESXi 6.5 during the vSAN On-disk Format version upgrade
  • After disk version upgrade, the following warning appears beside Disk format version under vSAN cluster > Configure > General:
    "Disk Format Version: All xx disk on version 5.0 but with old vSAN objects"
  • Errors found in the vCenter /var/log/vmware/vpxd/vpxd.log logfile:
Error Type: VsanFault 
Description Id: com.vmware.vsan.diskconvertion.tasks.conversion 

 INFO vsan-mgmt[DFC-domain-c12_f5583b58-8b35-11e8] [VsanVcDiskFormatConverterImpl::_UpdateProgress opID=f5583b58-8b35-11e8] Progress: 10
 INFO vsan-mgmt[DFC-domain-c12_f5583b58-8b35-11e8] [VsanVcDiskFormatConverterImpl::ConvertDisks opID=f5583b58-8b35-11e8] All online disk updates complete
 WARNING vsan-mgmt[DFC-domain-c12_f5583b58-8b35-11e8] [VsanVcDiskFormatConverterImpl::ConvertDisks opID=f5583b58-8b35-11e8] Exception in online upgrade: (vim.fault.VsanFault) {
  dynamicType = <unset>,
  dynamicProperty = (vmodl.DynamicProperty) [],
  msg = '',
  faultCause = <unset>,
  faultMessage = (vmodl.LocalizableMessage) [
     (vmodl.LocalizableMessage) {
        dynamicType = <unset>,
        dynamicProperty = (vmodl.DynamicProperty) [],
        key = 'com.vmware.vsan.diskconversion.msg.bumpupversionerror',
        arg = (vmodl.KeyAnyValue) [],
        message = 'Failed to bump up format version for diskmapping naa.xxxx, Failed to get VsanInfo operation lock for diskOpLockan operation is currently in progress(locked pid: 0), error: /tmp/.vsanDiskOpLock.lock.LOCK: timout waiting for lock after 30 seconds. Lock is currently held by process 838647 (python: python /usr/lib/vmware/vsan/perfsvc/vsanperfsvc.pyc)'
     }

Environment

VMware vSAN 6.6.x
VMware vSAN 6.5.x

Cause

  • In vSAN 6.7, the vCenter vpxd service upgrades the vSAN on disk format version per disk per group
  • When vSAN On-disk Format version on disk groups is being upgraded, the disk operation lock must be acquired to continue the upgrade operation, if this lock cannot be acquired within 30 seconds, the upgrade request will fail.
  • If a vSAN host has more than 1 disk group to be upgraded, there can be conflict on acquiring this lock, and if a disk operation lock takes more than 30s, other disk group requests on this host will fail causing the on disk upgrade operation to fail

Resolution

  • Upgrade all ESXi hosts to 6.7, it is advisable to have ESXi and vCenter Server on same release version before running an On-disk Format version upgrade
 


Additional Information

Following workaround is available  :
  • Login to vCenter Server & make a backup copy of this file:
cp /usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcDiskFormatConverterImpl.py /usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcDiskFormatConverterImpl.py.backup
  • Open file /usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanVcDiskFormatConverterImpl.py, go to LINE2725  
  • Change lines 
if (ssd.vsanDiskInfo and
     (ssd.vsanDiskInfo.formatVersion in ONLINE_UPGRADE_DISKGROUP_VERION)):

      WITH

      if (ssd.vsanDiskInfo and
      (ssd.vsanDiskInfo.formatVersion in ONLINE_UPGRADE_DISKGROUP_VERION) and
      (ssd.vsanDiskInfo.formatVersion != targetVersion)):
  • Restart vSAN health service via UI or through command on vCenter Server
       service-control --stop vmware-vsan-health
       service-control --start vmware-vsan-health
     


Impact/Risks:
  • Full features of v5 will not be available as the object will be on v3, but there is no potential risk to data


Powered by Wolken Software