New config vVols objects created on ESXi 8.0U1

Products

VMware vSphere ESXi

Issue/Introduction

This article discusses compatibility of Virtual Machines created on ESXi 8.0.U1 with older ESXi hosts. These Virtual machines disks reside on vVol Datastores.

Symptoms:

ESXi 8.0.U1, the new config vVol objects by default are created as 255 GB thin provisioned vVols instead of 4GB thin provisioned vVols. Also, the virtual disks created within these new config vVols, will not be backward compatible, i.e. older ESXi hosts will not be able to open these such virtual disks.

Relocating VMs created on ESXi 8.0.U1 having vVol backed virtual disk to an older ESXi host, fails with below error message:

Device 'Hard disk X' is not supported. "This is a general limitation of the virtual machine's compatibility with the ESXi version of the selected host."

A VM created (after upgrading some ESXi hosts to 8.0.U1) fails to power on.
A FCD catalog on vVol datastore (created after upgrading some ESXi hosts to 8.0.U1) may have FCDs that fail to attach to a running VM.
If a user is having disaster recovery solution like SRM, post recovery on the DR site VM may fail to Power-ON.

Environment

VMware vSphere ESXi 8.0.1

Cause

Config vVols act as sub-directories for a vVol datastore. Each Config vVol entry here is mounted individually. In earlier releases the size of these Config vVols is fixed at 4GB, which restricts the usage of these folders in a vVol datastore as content repositories.

To overcome this limitation Config vVols will now be created as 255-GB thin-provisioned objects. Additionally, VMFS-6 will be used to format these objects instead of VMFS-5. Lastly, sidecars in vVols in earlier releases were created as vVols objects, which introduced the overhead of VASA operations like bind/unbind on them as well as snapshots through VASA calls. Solutions like FCDs create a large number of small side cars, which caused performance issues on vVols when snapshots were taken. Since for each snap disk additional sidecar needs to be created, leading to vVol object creation while taking snapshot. This also consumed a large amount of vVols objects on storage targets without adding any value (since sidecars are small and the IO performance to them is not critical).

To improve performance of snapshots and scalability of vVol datastores, sidecars are now treated as files under Config vVols and normal file operations can be performed on them.

Resolution

To disable this feature, i.e. to ensure ESXi 8.0U1 host doesn't create 255 GB thin-provisioned vVols and instead falls back to older behavior of creating 4GB thin provisioned vVols, follow below steps:

Run the below command

esxcli system settings advanced set -o /VVOL/vvolUseVMFS6AndLargeConfigVVols -i 0

Use the below command to check the value:

> esxcli system settings advanced list -o /VVOL/vvolUseVMFS6AndLargeConfigVVols

                  Path: /VVOL/vvolUseVMFS6AndLargeConfigVVols
                  Type: integer
                  Int Value: 0 <<<<<<<<<<<<<------ Effective Value
                  Default Int Value: 1
                  Min Value: 0
                  Max Value: 1
                  String Value:
                  Default String Value:
                  Valid Characters:
                  Description: Use VMFS6 for config vVols and create
larger (255 GB) config vVols
                  Host Specific: false
                  Impact: none
          >

To convert a VM which has been created with 255 GB thin-provisioned vVol into a VM having 4 GB thin provisioned config vVol, follow below steps:

Run the below command to disable the feature on ESXi host:

esxcli system settings advanced set -o /VVOL/vvolUseVMFS6AndLargeConfigVVols -i 0

Relocate the VM into any other available datastore, i.e. perform a storage migration.
Relocate the VM back into vVol datastore.

How to identify VM using larger config vVols:

In order to support this change the version of a virtual disk has been bumped up, the version of the virtual disk can be obtained via CLI, MOB or VC API, in the VirtualDevice summary.

If the version of the virtual disk aka vDiskVersion is set to 7, then the VM is using larger config vVols and will not be compatible with (< 8.0U1 ESXi hosts)

From MOB:

ManagedObjectReference:VirtualMachine::VirtualMachineConfigInfo::VirtualHardware::VirtualDevice::vDiskVersion give the virtual disk version.

From esxcli:

> /bin/vim-cmd vmsvc/device.getdevices 1 | grep vDiskVersion
vDiskVersion = 7,
>

Note: Here the above VM is using newer/larger config vVols

Additional Information

Impact/Risks:
While this new functionality is introduced in the 8.0U1 release, it adds some restrictions in terms of how the VMs created by 8.0U1 hosts on a vVol datastore will work with older ESXi hosts.

VMware vCenter is aware of these limitations and will avoid starting a VM created with the new Config vVol scheme on an older ESXi host, but if all the ESX 8.0U1 hosts under vCenter are unavailable (down/unreachable for whatever reason) or if too few resources are available on these upgraded ESXi hosts the user may see the symptoms mentioned for this KB.

Note: For disaster recovery solution like SRM, on the DR site virtual disks created with in newer config vVols cannot be opened by older ESXi(<8.0U1) hosts. Hence it is advisable to upgrade ESXi hosts on the DR site as well.