VMware backups failing with Status 23 and Status 6 on Veritas.

Products

• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

• All VMware backups using either nbd transport mode or san transport mode are failing with Status 6 with backup job details in activity monitor showing message
  
  ERR - Error opening the snapshot disks using given transport mode: Status 23.
  
  
• The snapshots appear to be successful but the bpfis log and the snapshot job details showing the following messages
  
  09:13:09.051 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_Connect: Establish connection.
09:13:09.051 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Disable VMotion.
09:13:10.205 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: Licenses could not be retrieved for supplied user credentials.
09:13:10.439 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: VixDiskLib_PrepareForAccess returned:3014 [Insufficient permissions in the host operating system]
09:14:00.219 [9028.8020] <2> send_keep_alive: INF - sending keep alive
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: Calling VixDiskLib_PrepareForAccess
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_PrepareForAccess: Prepare to access disk.
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_Connect: Establish connection.
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Disable VMotion.
09:14:11.685 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: Licenses could not be retrieved for supplied user credentials.
09:14:11.919 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: VixDiskLib_PrepareForAccess returned:3014 [Insufficient permissions in the host operating system]

Environment

vCenter: 7.x,8.x

VCF: 5.x

Cause

• Several permissions were missing in the vCenter role used by the account specified for vCenter server credentials on the master server.
• This account was also not added to vCenter level permissions tab with propagate down checked.

Resolution

Modifying vCenter permissions is a VMware administration task, not a NetBackup function. The steps below are for reference only, coordinate with your VMware administrator before applying any changes.

1. Locate the account specified on the NetBackup master server as follows:

   1. Open the NetBackup Administration Console and go to "Media and Device Management -> Virtual Machine Servers -> Credentials"

   2. Double-click on the vCenter in question.

   3. Identify the username specified for this vCenter server.

2. Log into vCenter server using vSphere client.

3. In vSphere client go to "Home -> Administration -> Roles"

4. Identify the vCenter role used for the VMware backup agent account from (1) above.

5. Confirm this vCenter role has all of the permissions

6. If any required permissions are missing, add them to this role and save the changes.

7. Next, in vSphere client navigate to one of the vCenter inventory views under "Home -> Inventory." Either "Host and Clusters" or "VMs and Templates" will work.

8. In the left-hand pane, locate and click on the vCenter server in the topology.

9. Click on the vCenter server's "Permissions" tab.

10. Confirm that the account specified in (1) above is visible in the User/Group column.

11. If not this user should be added. Or be confirmed to be in a group that is added.

12. Double-click on the user or group.

13. Confirm it is in the correct "Role" that has the permissions defined in (5) above.

14. Confirm the "Propagate" box is checked.

 

 

Additional Information

The snapshot creation and removal processes are successful on vCenter; however, the task fails in Veritas.