vMotion of virtual machines fails with the error: "[msg.namespaceDb.badVersion] Incompatible version -1 (expect 2)"

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

Symptoms:
In a vSphere 6.5 U1, or vSphere 6.7 environment with NSX for vSphere 6.4.1 and above with Guest Introspection, you experience these symptoms:

• Migrating any new VMs may be impacted if the destination host has more than 170 VMs.
• The vmware.log file contains the following entries similar to:

2018-07-18T02:41:32.035Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2018-07-18T02:41:32.035Z| vmx| I125: Migrate: Caching migration error message list:
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.migration.failedReceive] Failed to receive migration.
2018-07-18T02:41:32.035Z| vmx| I125: [msg.namespaceDb.badVersion] Incompatible version -1 (expect 2).
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.mrestoregroup.failed] An error occurred restoring the virtual machine state during migration.

• In the /var/log/vmkernel.log file on the ESXi host, you see entries similar to:

2018-07-18T02:32:43.011Z cpu5:66134)WARNING: Heap: 3534: Heap fcntlHeap-1 already at its maximum size. Cannot expand.
2018-07-18T02:41:35.613Z cpu2:66134)WARNING: Heap: 4169: Heap_Align(fcntlHeap-1, 200/200 bytes, 8 align) failed. caller: 0x41800aaca9a3

Cause

The vMotions are failing because the fcntlHeap has been exhausted.

• On NSX for vSphere 6.4.1 and above, the user world component of Guest Introspection called EPSecMux aka Endpoint Multiplexor or Mux uses Namespace database to store its VM to solution (anti-virus) mapping.
• Namespaces will be created whenever a VM is created/moved to a host and deleted whenever it is removed/moved from a host. Also, whenever a VM connects, namespace database is read to restore protection to the VM immediately.
• Currently the namespace database is not big enough to handle large number of VMs moving to the host in a short interval of time such as during bulk vMotion.

Resolution

This issue is resolved for ESXi 6.5 in ESXi650-201811002, available at VMware Patch Downloads. 
This issue is resolved for ESXi 6.7 in ESXi 6.7u2, available at VMware Downloads