DRS migration fails with with error "Permission to perform this operation was denied. NoPermission.message.format"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• DRS assisted automated vMotion tasks will fail to balance the cluster with error "Permission to perform this operation was denied. NoPermission.message.format"
• Manual vMotion works fine to another ESXi host in the same cluster.
• No physical TPM/ TPM 2.0 is available on the ESXi host.
• Impacted VMs are identified to be configured on vTPM.
• encrypted vMotion is configured. Native KMS is configured.

Environment

VMware vCenter server 7.0.x

VMware vCenter server 8.0.x

Cause

/var/log/vmware/vpxd/vpxd.log

 

> ESXi host is connected to vCenter server with a NULL state.. and thus vMotion attempt fails.

> ESXi host seems like it is connected on vCenter server.. thou, its a 'false-positive' connection.. No true connection exists between the vCenter server and ESXi host.

> Refer below error snips for complete brevity on this context..

 

[YYYY-MM-DDTHH:MM:SS] warning vpxd[07187] [Originator@6876 sub=CryptoManager opID=lro-318337046-7323d5d9-01-01] The session <NULL> of user  does not have privilege Cryptographer.RegisterHost on entity [vim.HostSystem:host-331029, ESXi-host-FQDN].

 

[YYYY-MM-DDTHH:MM:SS] info vpxd[07224] [Originator@6876 sub=vpxLro opID=lro-318337043-576df78c-01-01] [VpxLRO] -- FINISH lro-318337045
[YYYY-MM-DDTHH:MM:SS] info vpxd[07224] [Originator@6876 sub=Default opID=lro-318337043-576df78c-01-01] [VpxLRO] -- ERROR lro-318337045 --  -- VmprovWorkflow: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    object = 'vim.HostSystem:f04251bc-6bf7-45f1-96f6-68e72235046c:host-331029',
-->    privilegeId = "Cryptographer.RegisterHost",
-->    missingPrivileges = <unset>
-->    msg = ""
--> }
--> Args:
-->
[YYYY-MM-DDTHH:MM:SS] error vpxd[07224] [Originator@6876 sub=drsExec opID=lro-318337043-576df78c-01] Failed migrating VM [vim.VirtualMachine:vm-338860,VM-Display-Name] to host vim.HostSystem:host-331029

[YYYY-MM-DDTHH:MM:SS] error vpxd[07224] [Originator@6876 sub=drsExec opID=lro-318337043-576df78c-01] Failed migrating VM [vim.VirtualMachine:vm-338269,VM-Display-Name] to host vim.HostSystem:host-331029

[YYYY-MM-DDTHH:MM:SS] warning vpxd[07187] [Originator@6876 sub=CryptoManager opID=lro-318337046-7323d5d9-01-01] The session of user  does not have privilege Cryptographer.RegisterHost on entity [vim.HostSystem:host-331029,ESXi-host-FQDN].

[YYYY-MM-DDTHH:MM:SS] error vpxd[07190] [Originator@6876 sub=MoDVSwitch opID=HB-host-331029@1007493-eeadef1] RPC for host [vim.HostSystem:host-331029,ESXi-host-FQDN] got status code 3

[YYYY-MM-DDTHH:MM:SS] error vpxd[07190] [Originator@6876 sub=MoHost opID=HB-host-331029@1007493-eeadef1] Null response from host [vim.HostSystem:host-331029,ESXi-host-FQDN]; no error

 

[root@ESXi_hostname:~] esxcli hardware trustedboot get
   Drtm Enabled: false
   Tpm Present: false

[root@ESXi_hostname:~] esxcli system settings encryption get
   Mode: NONE
   Require Executables Only From Installed VIBs: false
   Require Secure Boot: false

 

On the VM.vmx file, we could confirm the parameter  

vtpm.present = "TRUE"

Resolution

• We disconnected/ reconnected the ESXi hostname = 'ESXi-host-FQDN' on vCenter server inventory. 
• Further to a true connectivity now restored between vCenter server and impacted ESXi host.
• DRS recommended vMotion's are now working fine. Cluster DRS score has now increased to 80% ~ 100%.

Additional Information

The user will get an error prompt as " Cannot complete login due to an incorrect username or password " when user attempts to reconnect the ESXi host back in to vCenter server.. 

Administrator will have to provide the correct root credentials for the respective ESXi host to re-establish a true connectivity between vCenter server and ESXi host.