Virtual Machine backups may fail intermittently with an error "Snapshot creation is disabled by []"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• In the backup software logs, we may see an error similar to "[Snapshot creation is disabled by [vCloud Director or site recovery manager]"
• In the vCenter UI, we will find some of the options like Take snapshot, Delete all snapshots, Edit settings, Rename and Edit notes are greyed out for the affected VM
• In the vCenter logs and virtual machine tasks we do not see a create snapshot task for the affected VM.

Environment

VMware vCenter Server

Cause

As part of prechecks from the backup software, it will check if a snapshot of the VM can be taken or not. when the take snapshot option is disabled for the affected VM, the backup will fail with an error "Snapshot creation is disabled by [XXXX]"

Note: The [XXXX] portion of the error can be either blank or a product like vcloud director or site recovery manager. It is fetched from the value of managedby property of the VM

Additional information:

• VM actions like create snapshot can be disabled at runtime due to certain operations or can be explicitly disabled by some solution or user.
• To check for disabled actions of the VM, we can review the disabledmethod section of the VM mob page(https://<vc-ip>/mob/?moid=vm-XXXXX)
• If the action is disabled by a solution or user, it will also be listed in the database (select * from VPX_DISABLED_METHODS;)

Resolution

• For this specific case involving backups where the create snapshot method will be disabled at runtime which can be seen under disabledmethod section of the VM mob page but not in VPX_DISABLED_METHODS table in VCDB, engage the respective backup vendor

Note: Even though the create snapshot method is disabled at runtime, creating a snapshot using an API may succeed. Due to this nature of the issue, backup vendors have an option of disabling the pre-check for snapshot operations.

Workaround:

• If the VM action is only disabled at runtime(VM mob shows the disabled method but VCDB does not have any disabled methods for the VM), migrate the affected VM to a different ESXi host. After the migration all the disabled options should be active again and backup would succeed in the next run.
• If the VM action shows disabled in VCDB, this needs further investigation to identify which user or solution has disabled the action. This can also happen if there is any ongoing task on the VM which disables some methods

Note: The contents of VPX_DISABLED_METHODS table in VCDB may have information related to the source and reason for disabled actions.