VM deployment fails with "Unable to process template" in Content Library
search cancel

VM deployment fails with "Unable to process template" in Content Library

book

Article ID: 410651

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi VMware Live Recovery

Issue/Introduction

VM deployment from templates hosted in a Content Library fails during the compute resource selection phase. This issue typically occurs in environments utilizing **Storage DRS (SDRS)** and **vSphere Replication (VR)**. When stale replication data or orphaned RDID files exist on a datastore, SDRS flags the disks as "non-movable," preventing the Content Library from successfully processing the template deployment

Symptoms

  • Deploying a VM from the Content Library triggers a red error box in the vSphere Client: `Unable to process template`.

  • In `/var/log/vmware/content-library/cls.log`, the following exception is observed:
      com.vmware.vim.vmomi.core.exception.UnmarshallException: Missing value for non-optional field vm

  • In `/var/log/vmware/vpxd/vpxd.log`, Storage DRS faults are recorded:
     yyyy-mm-ddT error vpxd[####] [Originator@6876 sub=drmLogger opID=####] DrmFault: reason balanceDatastoreSpaceUsage, vm NotFoundObject, host NotFoundObject, fault [N3Vim5Fault27StorageDrsHbrDiskNotMovableE:####]
        yyyy-mm-ddT error vpxd[####] [Originator@6876 sub=drmLogger opID=####] DrmFault: reason balanceDatastoreSpaceUsage, vm NotFoundObject, host NotFoundObject, fault [N3Vim5Fault13RuleViolationE:####]

  •  In the vSphere Replication management logs located at /opt/vmware/hms/logs/hms.log , errors indicate a failure to retrieve DRM info for specific replication IDs:
         yyyy-mm-dd ERROR com.vmware.hms.replica.sdrs [tcweb-####] (..hms.sdrs.ReplicaMoveManagerImpl) [operationID=####,sessionID=####] | Could not retrieve drm info for disk RDID-#### , hbrServerUuid: #### , hbrGroupMoId: Hbr.Replica.Group.GID-####
        com.vmware.hms.sdrs.entities.DrmInfoRetrievalException: Could not retrieve drm info for disk RDID-####

Environment

  • vCenter 8.0.3
  • vCenter 9.x
  • VMware Live Recovery 9.0.x
  • ESXi: 8.0.3
  • ESX 9.x

Cause

The issue is caused by stale replication groups or orphaned RDID files remaining on the datastore. These usually persist following an improper replication stop or incomplete cleanup of deleted VMs. Storage DRS identifies these disks as replication-protected but cannot retrieve their metadata, leading it to mark them as "non-movable," which blocks the VM placement workflow.

Resolution

Option 1: Clean up Stale Replication Data (Recommended)

1. Identify Stale RDID Files: Use the `hms.log` snippets above to identify the specific `RDID-####` causing the `DrmInfoRetrievalException`.

2.  Verify VM Status: Determine if the VM associated with that RDID still exists in the replication inventory.

3.  Perform Cleanup:

  1. If the VM was previously removed, ensure it is also deleted from any Protection Groups.
  2. Remove any orphaned placeholder VMs on the recovery site.
  3. Manually delete the stale RDID files from the recovery datastore.Find the vmx files using find command from the datastore - find /vmfs/volumes/<datastore_name_or_uuid> -name "*.vmx"


Alternatively the Stale RDID files can be identified from the HMS Database dumps. Contact Broadcom Support for assistance with DB Queries

To access the vSphere Replication Management Server (HMS) database, you can use the built-in connection script or the standard PostgreSQL command-line tool.

Method 1: Built-in Connection Script

The easiest way to connect to the embedded HMS database is using the provided script, which automatically handles the connection parameters.

  1. SSH into the vSphere Replication appliance as root.
  2. Run the following command:  /opt/vmware/hms/bin/embedded_db_connect

Method 2: Manual psql Connection

If you need to connect manually (e.g., using the psql utility directly), follow these steps:

  1. Retrieve the DB password by viewing the configuration file: cat /opt/vmware/hms/conf/embedded_db.cfg
  2. Access the database using the vrmsdb user:
    /opt/vmware/vpostgres/current/bin/psql -U vrmsdb

Note: When prompted, enter the password found in step 1.

Critical Warnings and Caveats

  1. Stop the HMS Service First: Before performing any direct database modifications or troubleshooting tasks, it is strongly recommended to stop the HMS management service to prevent data corruption or service conflicts.
    Stop service: systemctl stop hms
    Start service: systemctl start hms
  2. Backups: Always take a snapshot of the vSphere Replication appliance before manually interacting with or modifying the database.
  3. Embedded Database Service: Ensure the Postgres service itself is running. If you cannot connect, you may need to restart it: systemctl restart hms-vpostgres


4.  Reconfigure Active Replications: If the VM exists but is in an error state, reconfigure the replication and ensure it completes a successful sync.

When reconfiguring active replications that are in an error state  ensure you observe the following warnings and prerequisites to prevent data loss or further synchronization failures:

Critical Warnings & Prerequisites

  1. Identify Target Datastores First: Before reconfiguring, you must note the target datastore of each VM and validate the target datastore seeds. Reconfiguring without this information can lead to selecting the wrong destination, which may result in a new full synchronization rather than using existing data note target datastore of each VM beforehand.
  2. Validate Target Seeds: Ensure that target datastore seeds are validated. If you configure a replication to use a VM with snapshots as a seed, the configuration might succeed, but the replication will enter an Error state at the end of the Initial Full Sync goes into the Error state at the end of the Initial Full Sync.
  3. Transient Error States: During the initial full synchronization after reconfiguration, the status may change temporarily to "Error" and back to normal multiple times. Do not interrupt the process unless the error persists state changes temporarily to Error and back to normal multiple times.
  4. Network Encryption Status: If your replications were originally configured with network encryption, they may enter a "Not Active" state after infrastructure changes. You must reconfigure them to restore connectivity reconfigure existing replications to use vSphere Replication with enhanced replication capabilities

Procedure Reminder

  1. Manual Synchronization Check: After the reconfiguration task is finished, you must manually ensure that all replications complete a successful synchronization to verify the error state is fully resolved ensure that all replications complete a successful synchronization.
  2. IP Address Mismatches: If the replication error is due to an IP address change (e.g., after an appliance upgrade or redeployment), ensure the source hosts are updated to use the new IP address of the target replication server during the reconfiguration source hosts use the new IP address of the target vSphere Replication server.
  3. Large Disk Resizing: If the VM enters an error state after a large disk (4TB+) was resized, reconfiguration is often required to recognize the new disk geometry VMs with large disks (4TB+) is resized the VM goes into an error state.


5.  Validation: Verify that `hms.log` no longer reports DRM info retrieval exceptions for those files.

Option 2: Disable Storage DRS

1.  Temporarily disable **SDRS** on the affected cluster.

2.  Restart the `vpxd` service on the vCenter Server.
    * Stopping, Starting or Restarting VMware vCenter Server Appliance services

Powered by Wolken Software