vSAN Data Protection Plugin displays "502 Bad Gateway" error on vCenter UI

Products

• VMware vSAN

Issue/Introduction

• After deploying the vSAN Data Protection appliance, the plugin in the vCenter UI displays a static "502 Bad Gateway (nginx/1.25.2)" error.

• The Data Protection management interface is inaccessible. 

• Snapshot operations cannot be managed or initiated via the UI.

• /var/log/vmware/snapservice/snap-service-panic.log show the panic event as below.

goroutine 602 [running]:
gitlab.eng.vmware.com/core-build/vsan_snapshot_service/pkg/server/vsphere/vcenter.GovmomiToVmDisk(...)
        /build/mts/release/bora-24755667/snapservice/pkg/server/vsphere/vcenter/server_vmomi_converter.go:106
panic: interface conversion: types.BaseVirtualDeviceBackingInfo is *types.VirtualDiskSparseVer2BackingInfo, not *types.VirtualDiskFlatVer2BackingInfo

Environment

• Product: VMware vSAN 8.x.

• Component: vSAN Data Protection Appliance / Snapshot Service.

Cause

• The vSAN Snapshot Service (snapservice) container fails to initialize because the one or more VMDK queries by vCenter are in an unsupported disk format (monolithicSparse or 2gbsparse). Specifically, if the VM was deployed or cloned from a source where the disk format became monolithicSparse (a format typically used by VMware Workstation or hosted products), the snapservice cannot mount the volume as a native vSAN object.

• Upon running a query to read all the VMDKs in the vCenter server, VMDKs may have createType as non VMFS.

[root@localhost] cat example.vmdk
# Disk DescriptorFile
version=4
encoding="UTF-8"
CID=f8292ff7
parentCID=ffffffff
createType="monolithicSparse" ------------> createType show monolithicSparse.

Resolution

To resolve this issue, identify the VMDK files in the vCenter inventory with non-vmfs createType and must be converted to a native ESXi format 'vmfs'.

To find the VMDK with non-vmfs createType, read the vmdk using method below.

1. Login to ESXi host SSH session.

2. Navigate to /vmfs/volumes.

3. Run command 'cat */*/*.vmdk' | grep createType' 

4. This is expected to read all the VMDKs in that respective ESXi hosts on all the datastores. However, if there are more clusters on vCenter, it has to be verified across the clusters.

To fix the identified vmdk in non-vmfs format, follow the instructions below.

Prerequisites:

• Ensure you have SSH access to the ESXi host where the VM with affected vmdk is registered.
• Identify the datastore path for the VM and VMDK.

Step-by-Step Instructions:

1. Power Off the VM with affected vmdk.
2. Verify Disk Format:
   • SSH to the ESXi host.
   • Navigate to the VM directory and run: grep "createType" <appliance_disk>.vmdk
   • If the output shows createType="monolithicSparse", proceed with conversion.
3. Convert the VMDK:
   • Run the vmkfstools command to clone the disk into a thin-provisioned vmfs format:
     bash

     vmkfstools -i "/vmfs/volumes/Datastore/example-vm/old-disk.vmdk" "/vmfs/volumes/Datastore/example-vm/new-disk.vmdk" -d thin

4. Update VM Settings:
   • In the vSphere Client, Edit Settings for the VM.
   • Remove the old Hard Disk (do not delete from disk yet).
   • Add a New Existing Disk and select the new-disk.vmdk created in step 3.
5. Finalize:
   • Power on the VM.
   • Verify the VM is healthy. Once confirmed healthy, you may delete the old monolithicSparse file.
6. Verify the vSAN Data protection plugin health once all the non-vmfs format disks are fixed to 'createType=vmfs'