ESXi host displays repeated 'Cannot login user [email protected]: no permission' events from HPE AMSDV or SUT agents in Lockdown Mode

Products

• VMware vSphere ESXi

Issue/Introduction

• You see high-frequency Cannot login user [email protected]: no permission events in vCenter's event viewer and in the affected host's /var/log/hostd.log.

• You see large volumes of vim.event.NoAccessUserEvent entries forwarded to vCenter from one or more ESXi hosts.

• The event rate is substantially higher than the every-five-minutes pattern described in the related vsan_health article. Rates of tens to hundreds of events per second have been observed when this scenario is active.

• In /var/log/hostd.log on the affected host, you see entries similar to:

  YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#####]: [Originator@#### sub=Solo.Vmomi] Activation finished; <<<session-id>, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : #####'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.version9, internal, 5.5>, ...
  YYYY-MM-DDTHH:MM:SS.###Z Db(167) Hostd[#####]: [Originator@#### sub=Solo.Vmomi] Arg userName:
  YYYY-MM-DDTHH:MM:SS.###Z Db(167) Hostd[#####]: --> "local-root"
  YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#####]: [Originator@#### sub=Solo.Vmomi] Throw vim.fault.NoPermission
  YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#####]: --> object = 'vim.Folder:ha-folder-root',
  YYYY-MM-DDTHH:MM:SS.###Z In(166) Hostd[#####]: --> privilegeId = "System.View",
  

  Two fields distinguish this variant from the standard vsan_health pattern: the submitted username is local-root (not root), and the SOAP API tuple is <vim.version.version9, internal, 5.5> (the legacy vSphere 5.5 namespace, not the current v8_0_3_0 namespace used by in-base ESXi callers).

• The opID field on the failed Login activations carries the prefix esxcli-NN-####, identifying the caller as the host's esxcli binary being forked locally by another process.

• The HPE AMSDV daemon family runs under watchdog supervision as four binaries (amsdv, smadv, ahsdv, smadvrev) when the HPE AMSDV component is installed via a custom image baseline. On hosts that also have HPE Smart Update Tools, the sut daemon contributes a smaller share of the same call pattern.

• On hosts where iLO connectivity has degraded, the AMSDV or SUT agents retry tightly and the event rate climbs further. In /var/log/amsdv.log or /var/log/sut.log, you see correlated errors similar to:

  DetectILO_ams failed with -50
  VNIC Connect Failure, Enable VNIC and restart Server
  Redfish Not connected
  

Additional symptoms reported

• You experience an unexpected ESXi host crash or hang; the host goes offline and requires a reboot to bring it back online.
• You attach the host log bundle for review to determine the root cause and whether the outage is hardware or software related, and during that log review you notice the high-frequency Cannot login user [email protected]: no permission entries and ask whether they relate to the outage.

Environment

• ESXi 8.0 Update 3 (or later patch)
• HPE ProLiant hardware with an HPE-customized image
• HPE AMSDV management agent or HPE Smart Update Tools (SUT) are installed
• Lockdown Mode is enabled in Normal or Strict mode.

Cause

The HPE AMSDV daemon family (amsdv, smadv, ahsdv, smadvrev) and the HPE SUT daemon poll the local hostd over loopback to collect hardware health and inventory data. Each poll fork-execs the host's esxcli binary, which then opens a SOAP session to hostd at 127.0.0.1:8307.

These sessions authenticate using the username local-root and the legacy vSphere 5.5 SOAP API namespace (<vim.version.version9, internal, 5.5>). When Lockdown Mode is enabled and root is not present on the host's Lockdown Mode Exception Users list, hostd denies every such session and emits the Cannot login user [email protected]: no permission event. The agents do not back off after the rejection, so the rejection rate matches the agents' polling cadence, producing a high-frequency event flood.

This is third-party daemon behavior. The HPE agents are not part of the base ESXi image, so ESXi patches that address the vsan_health version of the same symptom (documented in KB 378651) do not address this variant. Resolution requires action on the Lockdown configuration, on the HPE agents themselves, or on the iLO subsystem the agents talk to.

If iLO connectivity from the host's userspace HPE agents has degraded (for example, the iLO Channel Interface driver or the Virtual NIC is unreachable), the agents retry the polling cycle more aggressively, increasing the event rate substantially.

Resolution

Three remediation paths are available. They can be combined.

Path 1: Add root to the Lockdown Mode Exception Users list (recommended workaround)

1. In the vCenter inventory tree, select the target ESXi host.
2. Open the Configure tab.
3. Under System, select Security Profile.
4. Locate the Lockdown Mode panel and click Edit.
5. Open the Exception Users tab in the Lockdown Mode dialog.
6. Click Add User, enter root, and click OK.
7. Click OK to save the Lockdown Mode dialog.
8. Wait 10 to 15 minutes for the next agent polling cycle, then inspect the host's Monitor > Events view to confirm that Cannot login user [email protected]: no permission events have stopped.

For fleet-wide application, deploy the same change via a host profile or via a PowerCLI loop iterating the cluster's hosts. The change is reversible by removing root from the Exception Users tab. The root principal already holds the Admin role on each host independently of Lockdown Mode, so this change does not grant capability that did not already exist; the Exception Users list controls only the Lockdown gate.

Path 2: Disable or remove the HPE management daemons (requires HPE coordination)

1. Open an SSH session to the affected ESXi host.

2. Disable the AMSDV init script so it does not start at boot:

   chkconfig amsdv off
   

3. Stop the running daemon family:

   /etc/init.d/amsdv stop
   

4. If the host also has HPE Smart Update Tools and SUT is not required for the HPE management workflow, repeat steps 2 and 3 for sut.

5. Reboot the host. After the reboot, confirm that the daemons do not return and that the event flood does not resume.

6. With HPE's confirmation that the agents are not required for active inventory or health workflows, remove the corresponding HPE component VIBs (for example amsdvComponent) using esxcli software vib remove. This requires placing the host in Maintenance Mode and a follow-up reboot.

The AMSDV and SUT agents feed data to HPE iLO Amplifier Pack, HPE OneView, and the local iLO. If that data is being collected by another mechanism or is not actively used, removal is reasonable. If the data is in use, leaving the daemons disabled affects HPE health and inventory reporting and should be coordinated with HPE Support.

Path 3: Restore iLO connectivity from the host (complementary)

If /var/log/amsdv.log or /var/log/sut.log shows repeated DetectILO_ams failed with -50, VNIC Connect Failure, or Redfish Not connected entries, the agents are looping tightly because they cannot reach the iLO. Restoring iLO connectivity does not stop the polling but allows each cycle to complete cleanly, which substantially reduces the event volume without removing the agents. This work is HPE-side and typically involves the iLO Channel Interface (CHIF) driver on the host, the iLO Virtual NIC configuration, and iLO firmware. Engage HPE Support for the specific iLO model on the host.

Additional Information

• For the vsan_health internal-caller version of this symptom in non-HPE environments, see ESXi Host Displays Repeated 'Cannot Login User [email protected]: No Permission' Events After Upgrading to Version 8.0 U3 with Lockdown Mode Enabled.
• For the original HPE SUT-focused article, see Login Events recorded every five minutes on HPE Host.
• For background on the high-frequency esxcli execution pattern these agents create, see High Frequency Execution of esxcli Commands on ESXi.
• The root principal already holds the Admin role on the host independently of Lockdown Mode. Adding root to the Exception Users list re-enables Lockdown-protected local authentication paths for the agent caller without expanding the role granted.