"Metadata and system validation failed" message when attempting a file-based restore of vCenter Server
Article ID: 344773
Updated On:
Products
VMware vCenter Server
Issue/Introduction
- Restore of vCenter Server Appliance (VCSA) from a backup taken after patching may fail
- During stage 2 of the restore process the validation message shows "Metadata and system validation failed" along with the partitions that have been resized, which may include but is not limited to the following:
Error: Unknown system resource type lv_lvm_snapshot Error: Unknonw system resource type seat Error:Unknonw system resource type vtsb
- Checking the /var/log/vmware/applmgmt/applmgmt.log file on the vCenter Server Appliance shows an error similar to the following
[YYYY-MM-DDTHH:MM:SS] [1630]INFO:util.Common:Executing command: lsblk -o name,size,type -n -l -b --json.
[YYYY-MM-DDTHH:MM:SS] [1630]INFO:util.Common:System size info: {'memory': 21, 'lv_root_0': 48, 'vtsdb': 25, 'autodeploy': 10, 'imagebuilder': 10, 'archive': 50, 'updatemgr': 100, 'core': 50, 'vtsdblog': 15, 'swap1': 25, 'db': 10, 'lifecycle': 100, 'netdump': 1, 'log': 10, 'lv_lvm_snapshot': 200, 'seat': 25, 'dblog': 15, 'cpu': 4}
[YYYY-MM-DDTHH:MM:SS] [1630]ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Error: Unknown system resource type vtsdblog
[YYYY-MM-DDTHH:MM:SS] [1630]INFO:util.Common:System size info: {'memory': 21, 'lv_root_0': 48, 'vtsdb': 25, 'autodeploy': 10, 'imagebuilder': 10, 'archive': 50, 'updatemgr': 100, 'core': 50, 'vtsdblog': 15, 'swap1': 25, 'db': 10, 'lifecycle': 100, 'netdump': 1, 'log': 10, 'lv_lvm_snapshot': 200, 'seat': 25, 'dblog': 15, 'cpu': 4}
[YYYY-MM-DDTHH:MM:SS] [1630]ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Error: Unknown system resource type vtsdblog
[YYY-MM-DDTHH:MM:SS] [6554] ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Resize 'memory' from 12MB to 19MB
The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
- Here is the df -h command output of vCenter disk layout before and after upgrades, this clearly shows a change in the disk layouts
Before vCenter Upgrade to 8.0Update2
/dev/mapper/vtsdb_vg-vtsdb 49G 24K 47G 1% /storage/vtsdb
/dev/mapper/vtsdblog_vg-vtsdblog 25G 24K 24G 1% /storage/vtsdblog
/dev/mapper/vtsdblog_vg-vtsdblog 25G 24K 24G 1% /storage/vtsdblog
After vCenter Upgrade to 8.0Update2
/dev/mapper/vtsdblog_vg-vtsdblog 49G 33M 47G 1% /storage/vtsdblog
/dev/mapper/vtsdb_vg-vtsdb 25G 37M 24G 1% /storage/vtsdb
Environment
VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x
Cause
- Partitions on the VCSA is not matching with the backup copy causing the failure to restore
- This can occur if the deployment size selected during the restore process is smaller than the size contained in the backup. This can include if individual settings such as CPU, Memory or partition sizes have previously been manually increased.
- There may be other causes of this so please investigate the /var/log/vmware/applmgmt/applmgmt.log fully for each specific failure occurrence.
Resolution
- The issue is resolved in the release vCenter Server 8.0 Update 2a Build 22617221
- Release Notes
- VMware Cloud Foundation downloads
Workaround:
In order to resolve the issue, implement the below workaround
- Log in to the current VCSA via ssh and check the partition size using the below command
df -h
- Navigate to the file location on the backup server
- To determine the resource and partition sizes defined in the backup, check and compare the partition size on the VCSA with the backup-metadata.json file within the backup files. The details are available under the section "SizeInfo":
Sample: These values are for "DeploymentSize": "small"
"SizeInfo": {
"memory": 21,
"swap1": 25,
"lv_root_0": 48,
"lv_lvm_snapshot": 200,
"netdump": 1,
"db": 10,
"dblog": 15,
"updatemgr": 100,
"imagebuilder": 10,
"seat": 25,
"core": 50,
"vtsdblog": 15,
"autodeploy": 10,
"vtsdb": 25,
"lifecycle": 100,
"log": 10,
"archive": 50,
"cpu": 4
"SizeInfo": {
"memory": 21,
"swap1": 25,
"lv_root_0": 48,
"lv_lvm_snapshot": 200,
"netdump": 1,
"db": 10,
"dblog": 15,
"updatemgr": 100,
"imagebuilder": 10,
"seat": 25,
"core": 50,
"vtsdblog": 15,
"autodeploy": 10,
"vtsdb": 25,
"lifecycle": 100,
"log": 10,
"archive": 50,
"cpu": 4
Note: For other deployment types, refer to the table below
- Modify the disk sizes under "
backup-metadata.json" to match with the deployed size - On the newly deployed VCSA any incorrectly sized resources or partitions would need to be increased to at or above the values in the json file by following the steps in the KB Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0
- Re-initiate the restore operation
NOTE:-
- Restore of vCenter Server Appliance (VCSA) from backup taken after patching to vCenter 8.0U2 may fail
- If the disk size was manually increased on the backup vCenter, then the sizes in the backup-metadata.json is correct, in that case one must increase the restore vCenter disk size to mirror the backup vCenter Disk size
- If the Restored vCenter has wrong disk mapping, then change in the backup-metadata.json will not help. One should increase the restored vCenter Disk size to under provisioned disks listed in the table above
Additional Information
For more information on deployment sizing see Hardware Requirements for the vCenter Server Appliance and Storage Requirements for the vCenter Server Appliance sections of the vCenter Server Installation and Setup guide.
Impact/Risks:
Unable to use existing backups to complete file based restore.
Feedback
Yes
No
Powered by
