Replicating or Recovering VMs with SR-IOV Adapters in VMware Site Recovery Manager (SRM)
Article ID: 412604
Updated On:
Products
Issue/Introduction
When attempting to replicate or recover virtual machines (VMs) that are configured with SR-IOV (Single Root I/O Virtualization) adapters, users may encounter challenges. SR-IOV provides direct network access to VMs for enhanced performance but introduces limitations that affect replication and live migration functionality.
Symptoms:
- SRM recovery workflow completes but fails during VM power-on.
- Manual VM power-on attempt at the recovery site fails with the same error, confirming the problem occurs post-recovery during PCI passthrough validation.
- In successful DR tests, the power-on task was skipped, while in failed tests, the power-on task was attempted, resulting in incompatibility.
- Dr.logs Events:
info vmware-dr[459403] [SRM@6876 sub=DrTask ctxID=41050b36 opID=7f378432-c7eb-46ae-bf9a-21d9c841361d-test:e3fa] Task 'dr.recovery.RecoveryManager.test1039' failed
--> (vim.fault.NoCompatibleHost) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> host = (vim.HostSystem) [
--> 'vim.HostSystem:xxxxxxxx-xxxxxxxx-xxxxxxxx-xxxxxxxx-xxxxxxxx:host-XXXXXX'
--> ],
--> error = (vmodl.MethodFault) [
--> (vim.fault.InsufficientResourcesFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> key = "com.vmware.vim.vpxd.vmcheck.assignHwNotAvailable",
--> arg = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "host",
--> value = "targethostfqdn"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "vm",
--> value = "sla71360"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "missing",
--> value = "pciPassthruXX"
--> }
--> ],
--> message = "One or more devices (pciPassthruXX) required by VM VM_NAME are not available on host targethostfqdn."
--> }
--> ]
--> msg = "Insufficient resources."
--> }
--> ]
--> msg = "No host is compatible with the virtual machine."
--> }
Environment
VMware vSphere Replication 8.x
VMware vSphere Replication 9.x
VMware Live Site Recovery 8.x
VMware Live Site Recovery 9.x
Cause
SR-IOV works by exposing multiple Virtual Functions (VFs) from a Physical Function (PF) network adapter, and directly assigning a VF to a VM. While this improves performance, it comes with limitations:
-
VF devices do not have permanent MAC addresses and are re-assigned after each host reboot.
-
SR-IOV passthrough mode bypasses vSwitch functionality, making the VM tightly coupled to the underlying host hardware.
-
Because of direct passthrough dependency, replication of SR-IOV passthrough adapters is not supported.
-
VF passthrough mode is incompatible with live migration , which is required for many advanced features like HA, DRS, and Live Recovery.
-
The failure is caused by the use of SR-IOV passthrough adapters within an SRM recovery workflow:
-
SR-IOV Virtual Functions (VFs) are tied to the physical host hardware.
-
The SR-IOV VF device required by the VM is unavailable on the recovery host, causing the incompatibility error.
-
Resolution
To replicate or migrate a VM with SR-IOV adapters in VMware Live Recovery:
-
Unplug the SR-IOV device from the VM on the source host before initiating replication or migration.
-
Perform the replication, recovery, or migration operation.
-
Reassign the SR-IOV VF on the destination host after the VM is successfully recovered or migrated.
Additional Information
Important Notes:
-
SR-IOV passthrough adapters cannot be replicated as part of VMware Live Recovery.
-
If uninterrupted mobility or disaster recovery is required, consider using standard VMXNET3 virtual adapters instead of SR-IOV for affected VMs.
-
Plan SR-IOV usage for performance-sensitive workloads where mobility is not a primary requirement.
SR-IOV Features and Limitations:
-
Granularity: Limited to predefined Virtual Functions (VFs).
-
Scalability: Fixed number of VFs, hardware-dependent.
-
Flexibility: Low flexibility as VFs are static.
-
Configuration Ease: Easier to configure, widely supported.
-
Adoption Status: Mature and standardized.
-
Use Cases: Performance-sensitive workloads in traditional virtualization, not recommended for environments requiring frequent migration or replication.
Key Consideration:
SR-IOV provides significant performance benefits but at the cost of VM mobility and disaster recovery features. Organizations must weigh these trade-offs when designing environments that rely on VMware Live Recovery.
Feedback
