"Metadata and system validation failed" message when attempting a file-based restore of vCenter Server post upgrade to 8.0U2

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Restore of vCenter Server Appliance (VCSA) from a backup taken after patching to 8.0U2 may fail
• /var/log/vmware/applmgmt/applmgmt.log

2023-10-21T13:32:37 PM UTC [1630]INFO:util.Common:Executing command: lsblk -o name,size,type -n -l -b --json.
2023-10-21T13:32:37 PM UTC [1630]INFO:util.Common:System size info: {'memory': 21, 'lv_root_0': 48, 'vtsdb': 25, 'autodeploy': 10, 'imagebuilder': 10, 'archive': 50, 'updatemgr': 100, 'core': 50, 'vtsdblog': 15, 'swap1': 25, 'db': 10, 'lifecycle': 100, 'netdump': 1, 'log': 10, 'lv_lvm_snapshot': 200, 'seat': 25, 'dblog': 15, 'cpu': 4}
2023-10-21T13:32:37 PM UTC [1630]ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Error: Unknown system resource type vtsdblog

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

• Here is the df -h command output of vCenter disk layout before and after upgrades, this clearly shows a change in the disk layouts

Before vCenter Upgrade to 8.0Update2 

/dev/mapper/vtsdb_vg-vtsdb       49G 24K 47G 1% /storage/vtsdb
/dev/mapper/vtsdblog_vg-vtsdblog 25G 24K 24G 1% /storage/vtsdblog

After vCenter Upgrade to 8.0Update2 
/dev/mapper/vtsdblog_vg-vtsdblog 49G 33M 47G 1% /storage/vtsdblog
/dev/mapper/vtsdb_vg-vtsdb       25G 37M 24G 1% /storage/vtsdb

Environment

VMware vCenter Server 8.0.2

Cause

• Partitions on the VCSA is not matching with the backup copy causing the failure to restore

Resolution

• The issue is resolved in the release vCenter Server 8.0 Update 2a Build 22617221
• Release Notes 
• Customer Connect Download Link

Workaround:
In order to resolve the issue, implement the below workaround

• Log in to the current VCSA via ssh and check the partition size using the below command

df -h

• Navigate to the file location on the backup server
• Compare the partition size on the VCSA with the backup-metadata.json file within the backup files. The details are available under the section "SizeInfo": 

Sample: These values are for "DeploymentSize": "small"

"SizeInfo": {
    "memory": 21,
    "swap1": 25,
    "lv_root_0": 48,
    "lv_lvm_snapshot": 200,
    "netdump": 1,
    "db": 10,
    "dblog": 15,
    "updatemgr": 100,
    "imagebuilder": 10,
    "seat": 25,
    "core": 50,
    "vtsdblog": 15,
    "autodeploy": 10,
    "vtsdb": 25,
    "lifecycle": 100,
    "log": 10,
    "archive": 50,
    "cpu": 4
 

Note: For other deployment types, refer to the table below 

• Modify the disk sizes under "backup-metadata.json" to match with the deployed size
• Re-initiate the restore operation

NOTE:- 

• Restore of vCenter Server Appliance (VCSA) from backup taken after patching to vCenter 8.0U2 may fail
• If the disk size was manually increased on the backup vCenter, then the sizes in the backup-metadata.json is correct, in that case one must increase the restore vCenter disk size to mirror the backup vCenter Disk size
• If the Restored vCenter has wrong disk mapping, then change in the backup-metadata.json will not help. One should increase the restored vCenter Disk size to under provisioned disks listed in the table above