Storage vMotion migration fails with the error: The method is disabled by 'SYMC-INCR dd-mm-yyyy hh:mm'

Products

• VMware vCenter Server

Issue/Introduction

• Restore of VM from 3rd party backup solution not completing (e.g., Dell Avamar to Cloud Director [VCD], etc.).
• Storage vMotion migration fails.
• VM Migration option is unavailable.
• Cannot drag and drop the virtual machine from one folder to another.
• Cannot move virtual machine between folders.
• The virtual machine Remove from Inventory option is unavailable.
• In vSphere Client, you see these errors:
   
  • The method is disabled by '_DP-VEPA_BACKUP_RESTORE_LOCK_'
Error Stack
Call "VirtualMachine.Relocate" for object "VM-NAME" on vCenter Server "vCenter-Name" failed
 
  • The method is disabled by 'SYMC-INCR dd-mm-yyyy hh:mm'
Error Stack
Call "VirtualMachine.Relocate" for object "VM-NAME" on vCenter Server "vCenter-Name" failed
 
  • The method is disabled by 'Data Recovery'
 
  • The method is disabled by 'vSphere Data Protection'
 
  • This method is disabled by 'moref=vm-ID'
 
  • Error:fault.MethodDisabled.summary
     
• Storage vMotion fails with the generic error message:
  
  The method is disabled by XXXXXXXXXXX
   
• When vMotion starts, this message appears:
  
  Invalid / explicitly disabled state operation is invoked

Cause

This issue occurs because the entries from the VPX_DISABLED_METHODS table are not removed after the virtual machine backup task completes. When a virtual machine level backup begins, the backup system informs vCenter Server to disable Storage vMotion for that virtual machine to ensure that the backups can complete successfully. If no backups are active for that virtual machine when this error occurs, Storage vMotion does not automatically re-enable.

Resolution

Please work with 3rd party products Backup vendor to fix their code to cleanly handle enable/disable methods API.
Note: Do not proceed with the following steps until it is confirmed the backup vendor has reviewed the situation with no resolution provided.

Caution: This procedure modifies the database. Ensure to take a full backup of the existing database before proceeding with any of these options. Involve a Database Administrator if there is any difficulty with these steps.

To work around this issue, use one of these options:

• Schedule another backup
• Manually remove entries from the vCenter Server database
• Manually remove entries from the vCenter Server Appliance vPostgres database
• Remove and re-add the virtual machine from the inventory
• Remove and re-add the ESXi/ESX hosting the virtual machine from the inventory

Schedule another backup

Schedule another backup of the affected virtual machine. If this completes successfully, the entries should be removed from the vCenter Server database and this allows a Storage vMotion of the virtual machine.

Manually remove entries from the vCenter Server SQL database

To manually remove the entries from the vCenter Server database:
 

Caution: This procedure requires temporarily disabling access to VC vSphere client and modifies the VCSA database. Ensure to coordinate with environment team and take a full backup of the existing database before proceeding with this workaround. Involve a Database Administrator if there is any difficulty with these steps.

1. Take proper snapshot of VCSA VM. If ELM (linked mode) take offline snapshots of all linked VCSA VMs. Otherwise take snapshot (no memory).
2. Connect to VCSA via SSH as root user.
3. Stop the vCenter Server "vmware-vpxd" service.
   
   service-control --stop vmware-vpxd
   
   
4. Connect to VCDB with “/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres“
5. Run SQL statement against the vCenter Server database:
   
   select * from VPX_VM WHERE FILE_NAME LIKE '%Virtual-Machine-Name%'
   
   where Virtual-Machine-Name is the name of the virtual machine. Leave the percentage (%) signs flanking the name.
    
6. Make a note of the vm-ID_Number value returned by the SQL statement in step 3, as this will be required in the SQL statement in steps 5 and 6.
7. To confirm the stale entry, run this SQL statement against the vCenter Server database:
   
   select * from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-ID from step 4'
    
8. To remove the stale entry, run this SQL statement against the vCenter Server database:
   
   delete from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-ID from step 4'
    
9. Start the vCenter Server "vpxd" service.
   
   service-control --start vmware-vpxd
10. Retry the Storage vMotion or restore of VM.

 

Remove and re-add the virtual machine from the inventory

To remove and re-add the virtual machine from the inventory:

Caution: Removing a virtual machine from inventory removes any performance statistics and historical data associated with the virtual machine. This process should only be used if the preceding procedures are unsuccessful or if any historical or performance data for a defined virtual machine.

 

1. Power off the virtual machine.
2. In vSphere Client, right-click the virtual machine and click Remove from Inventory.
3. Browse to the datastore where the virtual machine resides.
4. Right-click the .vmx file of the virtual machine and click Add to Inventory.
   
   Note: This gives the virtual machine a new ID.
    
5. Power on the virtual machine and retry the Storage vMotion.

Remove and re-add the ESX/ESXi from the inventory

Caution: Removing an ESX/ESXi host from inventory removes any performance statistics and historical data associated with the host. This process should only be used if the preceding procedures are unsuccessful or if no historical or performance data for the defined ESX/ESXi host needs to be retained.

1. In the vSphere Client, right-click the ESXi/ESX host and click Disconnect.
2. Right-click the disconnected ESXi/ESX host and click Remove from Inventory.
3. After the host is removed from inventory, right-click the cluster and click Add Host.
4. Provide the IP address or hostname of the ESXi/ESX host and the root username and password.

Additional Information

• When Storage DRS is enabled for a datastore cluster, it might be disabled on some virtual disks in the datastore cluster. This may result in the errors similar to those under symptoms section. For more information, see the Storage DRS is Disabled on a Virtual Disk section in the Storage DRS is Disabled on a Virtual Disk.

Enabling a virtual NIC for a virtual machine in a vDS portgroup after a storage migration fails: "Invalid configuration for device '0'" error when re-connecting VM's network adapter.