File-Based Backup and Restore of vCenter Server failes with "BackupManager encountered an exception"

Products

• VMware vCenter Server

Issue/Introduction

After upgrading to vCenter Server 7.0 Update 3, manual and scheduled backups to an NFS target fail.

In vCenter appliance's vmware.log and console screen, the following call stack is outputted:

YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  _pnfs_return_layout+0xe4/0x290 [nfsv4]
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  ? nfs_inode_detach_delegation+0x25/0x60 [nfsv4]
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  nfs4_evict_inode+0x4f/0x70 [nfsv4]
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  evict+0xbe/0x180
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  dispose_list+0x34/0x50
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  evict_inodes+0x13e/0x150
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  generic_shutdown_super+0x3f/0x110
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  nfs_kill_super+0x1c/0x40 [nfs]
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  deactivate_locked_super+0x43/0x70
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  deactivate_super+0x3b/0x50

YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  cleanup_mnt+0x3a/0x90
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  __cleanup_mnt+0xd/0x10
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  task_work_run+0x92/0xb0
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  exit_to_usermode_loop+0xd5/0xe0
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  do_syscall_64+0xfb/0x110
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
YYYY-MM-DD THH:MM:SS.mmmZ| vcpu-2| I125: Guest: <4>[XXXXX.XXXXXX]  RIP: 0033:0x7f5d5b4e7627

In vCenter backup.log, the following error is outputted:

YYYY-MM-DD THH:MM:SS.mmm [YYYYMMDD-HHMMSS-XXXXXXXX] [MainProcess:PID-35988] [MountedFSStorageIOLib::unmount:MountedFSStorageIOLib.py:276] ERROR: Failed to unmount the nfs share at /storage/remote/backup/nfs/<BACKUP_IP>/<ID>/<ID>, with flags: ; Err- rc: 255, out: , err:
YYYY-MM-DD THH:MM:SS.mmm [YYYYMMDD-HHMMSS-XXXXXXXX] [MainProcess:PID-35988] [BackupManager::main:BackupManager.py:659] ERROR: BackupManager encountered an exception: Plugin error occurred. ErrCode: 153, Args: ()

Environment

VMware vCenter Server 7.0 Update 3

Cause

This issue is caused by Linux kernel used in Photon OS for vCenter Server 7.0 U3. A NULL pointer dereference occurs during the layout return process when unmounting an NFS4 share, causing the umount.nfs4 command to fail.

Resolution

This issue is resolved in vCenter Server 7.0 Update 3d or the later build.

Workaround:

Use NFSv3: Change the version from NFSv4 to NFSv3 at NFS storage. Internal testing confirms that NFSv3 is not affected by this kernel bug.
Alternative Protocols: Use other supported protocols such as SMB, HTTP, HTTPS, FTP and FTPS for vCenter backups.

Additional Information

Japanese KB : vCenter Server のファイルベースのバックアップとリストアが "BackupManager encountered an exception" で失敗する