vMotion fails with error "NamespaceMgr could not lock the db file"

Products

• VMware vSphere ESXi

Issue/Introduction

vMotion fails intermittently with below error:

Failed to receive migration. The source detected that the destination failed to resume.
An error occurred restoring the virtual machine state during migration. NamespaceMgr could not lock the db file.

The destination ESXi host vmware.log shows the following messages:

[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.checkpoint.migration.failedReceive] Failed to receive migration.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.namespaceMgr.noLock] NamespaceMgr could not lock the db file.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.checkpoint.mrestoregroup.failed] An error occurred restoring the virtual machine state during migration.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - Msg_Post: Error
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.checkpoint.mrestoregroup.failed] An error occurred restoring the virtual machine state during migration.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.namespaceMgr.noLock] NamespaceMgr could not lock the db file.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - [msg.checkpoint.migration.failedReceive] Failed to receive migration.
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - ----------------------------------------
[YYYY-MM-DDTHH:MM:SS] In(05) vmx - Module 'CheckpointLate' power on failed.

Note: To identify the correct vmware.log on the destination host, see Locating virtual machine log files on an ESXi host.

The issue only impacts VM's provisioned on shared VMFS datastores, vSAN and VVOL that use a namespace DB. This issue does not impact VMs on NFS datastores.
A namespace DB will be used under certain environment conditions such as SDMP from vROps, guest introspection for NSX, etc.. 

Environment

VMware ESXi 8.0 U2b or later(23305546)
VMware ESXi 7.0 U3q (23794027)

Cause

The cause of the issue is due to locking mechanism on the namespace db and has been primarily observed on VM hardware versions <= 19.

Resolution

To resolve the issue upgrade the VM Compatibility to Hardware Version 19 (or the highest version supported by the ESXi host):)

1. Log in to the vCenter Server.
2. Create a backup or snapshot of the virtual machine.
3. Power off the virtual machine.
4. Click the Updates tab.
5. In the VM Hardware Compatibility panel, click Upgrade to Match Host.
6. Power on the virtual machine.

Reference KB- Upgrading a virtual machine to the latest hardware version

Note: The issue has been fixed in VMware ESXi 7.0 U3r and 8.0 U3e

Workaround

The issue is intermittent in nature and there is no reliable/permanent workaround known for this. Attempt one of the two workarounds:

1. If vMotion is failing consistently for a VM, try suspending/resuming or power cycling the VM to break the failure cycle. It might resolve the issue temporarily and may need to be repeated again in future if vMotion starts failing consistently for the VM again.
2. For intermittent vMotion failures for a VM, retrying vMotion is recommended.

The Knowledge Base article is attached with a script that checks the environments that are likely to encounter vMotion failure with error "NamespaceMgr could not lock the db file".

NOTE:

The ESXi host version check performed by this script may not provide reliable information for hot patches. Therefore, impacted version verification with Broadcom support is recommended for hot patches.

Steps to Execute the script

1. This script is tested on the following Microsoft PowerShell version and VMware PowerCLI version:
   PS C:\Users\TestUser> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
  5      1      22621  4391

PS C:\Users\TestUser> Get-Module -Name VMware.PowerCLI -ListAvailable
Directory: C:\Users\TestUser\Documents\WindowsPowerShell\Modules

ModuleType Version    Name     ExportedCommands
---------- -------    ----     ----------------
Manifest   12.4.0.... VMware.PowerCLI

To uninstall old version of VMware PowerCLI, run:

(Get-Module -Name VMware.PowerCLI -ListAvailable).RequiredModules | Uninstall-Module -Force
Get-Module -Name VMware.PowerCLI -ListAvailable | Uninstall-Module -Force
   
   
2. To install the latest VMware PowerCLI from PowerShell repository 'PSGallery', run:
   
   Install-Module -Name VMware.PowerCLI -Scope CurrentUser
This installs VMware PowerCLI to the current user directory $home\Documents\PowerShell\Modules, this will avoid running the script in an Admin PowerShell session.
   
   
3. You may also need to run this command before running any script:
   
   Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
   
   
4. Run this command to set Prompt if the server certificate is not trusted:
   
   Set-PowerCLIConfiguration -Scope Session -InvalidCertificateAction Prompt
   
   
5. Run the script from its full path :- For example if the script is located in the \tmp directory the run the command as below
   
   Command :- .\tmp\nsmgr-vmotion-precheck.ps1
   
   
6. To check an existing vCenter server for impacted ESXi hosts and VMs, run:
   
   .\<directory location of the script>\nsmgr-vmotion-precheck.ps1
.\<directory location of the script>\nsmgr-vmotion-precheck.ps1 -CheckVMsOnly (to check the impacted VMs only)

Executing the script, will ask for the vCenter Server IP address, User name (e.g. [email protected]) and the password credentials for the vCenter Server to validate the impacted hosts and VMs.