"Metadata and system validation failed" message when attempting a file-based restore of vCenter Server
search cancel

"Metadata and system validation failed" message when attempting a file-based restore of vCenter Server

book

Article ID: 344773

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Restore of vCenter Server Appliance (VCSA) from a backup taken after patching may fail.
  • During stage 2 of the restore process the validation message shows "Metadata and system validation failed" along with the partitions that have been resized, which may include but is not limited to the following:

     Error: Unknown system resource type lv_lvm_snapshot Error: Unknown system resource type seat Error:Unknown system resource type vtsb 

  • Checking the /var/log/vmware/applmgmt/applmgmt.log file on the vCenter Server Appliance shows an error similar to the following:
     [YYYY-MM-DDTHH:MM:SS] [1630]INFO:util.Common:Executing command: lsblk -o name,size,type -n -l -b --json.
     [YYYY-MM-DDTHH:MM:SS] [1630]INFO:util.Common:System size info: {'memory': 21, 'lv_root_0': 48, 'vtsdb': 25, 'autodeploy': 10, 'imagebuilder': 10, 'archive': 50, 'updatemgr': 100, 'core': 50, 'vtsdblog': 15, 'swap1': 25, 'db': 10, 'lifecycle': 100, 'netdump': 1, 'log': 10, 'lv_lvm_snapshot': 200, 'seat': 25, 'dblog': 15, 'cpu': 4}
     [YYYY-MM-DDTHH:MM:SS] [1630]ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Error: Unknown system resource type vtsdblog
     [YYY-MM-DDTHH:MM:SS] [6554] ERROR:backupRestoreAPI:Metadata and system validation failed; Err: Resize 'memory' from 12MB to 19MB 
  • Here is the df -h command output of vCenter disk layout before and after upgrades, this clearly shows a change in the disk layouts
         Before vCenter Upgrade to 8.0 Update 2 
    /dev/mapper/vtsdb_vg-vtsdb       49G 24K 47G 1% /storage/vtsdb
        /dev/mapper/vtsdblog_vg-vtsdblog 25G 24K 24G 1% /storage/vtsdblog

        After vCenter Upgrade to 8.0 Update 2 
    /dev/mapper/vtsdblog_vg-vtsdblog 49G 33M 47G 1% /storage/vtsdblog
    /dev/mapper/vtsdb_vg-vtsdb       25G 37M 24G 1% /storage/vtsdb
 
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

  • Partitions on the VCSA is not matching with the backup copy causing the failure to restore.
  • This can occur if the deployment size selected during the restore process is smaller than the size contained in the backup. This can include if individual settings such as CPU, Memory or partition sizes have previously been manually increased.
  • There may be other causes of this so please investigate the /var/log/vmware/applmgmt/applmgmt.log fully for each specific failure occurrence.

Resolution

The underlying Linux kernel issue where VMDKs were associated with incorrect mount points during deployment was resolved in vCenter Server 8.0 Update 2a (Build 22617221). Fresh deployments on or after this build do not experience random VMDK mount point swapping.

Restore Behavior in Subsequent Builds (Including vCenter Server 8.0 Update 3):
Deploying an appliance using vCenter Server 8.0 Update 2a, 8.0 Update 3, or later versions does not remediate pre-existing backups. If a backup was generated from an appliance running with swapped mount points or custom expanded partitions, backup-metadata.json retains those non-standard partition sizes. Stage 2 restore validation will continue to fail across all 8.0.x releases until the target virtual machine partitions match or exceed the metadata specifications.

Implement the workaround detailed below to align the target appliance partition configuration with the backup metadata.

Workaround

To resolve the validation failure during Stage 2, align the target appliance disk allocation with the values defined in backup-metadata.json.

Step 1: Enable SSH on the target (Stage 1) appliance

SSH is disabled by default on newly deployed appliances. Shell access must be enabled via the appliance console:

  1. Open the VMware Host Client or vSphere Client managing the ESXi host hosting the target appliance.
  2. Click "LAUNCH WEB CONSOLE" to open appliance console for the newly deployed appliance.
  3. Press F2 and authenticate using the root credentials configured during Stage 1.
  4. Navigate to Troubleshooting Mode Options > Enable SSH and press Enter.
  5. Verify the console confirms that SSH is enabled.

Step 2: Determine partition requirements from backup-metadata.json

  1. Navigate to the directory containing the vCenter backup on the backup storage repository.
  2. Open backup-metadata.json in a text editor and review the SizeInfo section corresponding to the deployment size.
    Sample: These values are for "DeploymentSize": "small"
    "SizeInfo": {
    "memory": 21,
    "swap1": 25,
    "lv_root_0": 48,
    "lv_lvm_snapshot": 200,
    "netdump": 1,
    "db": 10,
    "dblog": 15,
    "updatemgr": 100,
    "imagebuilder": 10,
    "seat": 25,
    "core": 50,
    "vtsdblog": 15,
    "autodeploy": 10,
    "vtsdb": 25,
    "lifecycle": 100,
    "log": 10,
    "archive": 50,
    "cpu": 4
}
  3. Identify any partitions with inverted or unusually high values (such as swapped allocations between vtsdb and vtsdblog).

    Note: For other deployment types, refer to the table below 

Step 3: Compare target appliance partitions against the metadata

  1. Connect to the target appliance via SSH as root.
  2. Execute the following command to display the disk layout and mount points:
    lsblk -o NAME,SIZE,MOUNTPOINT
  3. Cross-reference the output against backup-metadata.json to identify any partition on the target appliance smaller than the size listed in the backup metadata.

Step 4: Expand under-provisioned disks on the target appliance

Stage 2 validation requires every target partition to be greater than or equal to the size declared in backup-metadata.json.

  1. Edit the virtual machine settings of the target appliance in the ESXi or vSphere client.
  2. Locate the specific Hard Disk (VMDK) corresponding to the under-provisioned mount point and increase the provisioned size to match or exceed the JSON value (Refer to Broadcom KB Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0).
  3. In the SSH session on the target appliance, expand the logical volume:
    /usr/lib/applmgmt/support/scripts/autogrow.sh

Step 5: Resume Stage 2 restore

  1. Access the vCenter Server Appliance Management Interface (VAMI) at https://<Target-Appliance-IP>:5480/.
  2. Resume Stage 2 of the restore process. Stage 2 pre-check validation will complete successfully.

Additional Notes:

  • Editing backup-metadata.json directly: If the source backup contains swapped mount points (such as vtsdb and vtsdblog), the JSON file may alternatively be edited directly on the backup server to match the standard deployment size before initiating Stage 2, provided backup integrity checks or checksum mechanisms do not prevent file modification.
  • Compute Resource Mismatches: If the validation error specifies an insufficient cpu or memory allocation, adjust the target virtual machine CPU or RAM settings in the virtual machine configuration prior to resuming Stage 2.

Additional Information

For more information on deployment sizing see Hardware Requirements for the vCenter Server Appliance and Storage Requirements for the vCenter Server Appliance sections of the vCenter Server Installation and Setup document.

Impact/Risks:
Unable to use existing backups to complete file based restore.

This issue has also been seen due to an incorrect FQDN used for the restored vCenter name.

Powered by Wolken Software