Cannot login user [email protected] no permission errors every 5 minutes in NSX lockdown mode environments

Products

• VMware vSphere ESXi

Issue/Introduction

• NSX environments with ESXi lockdown mode enabled experience continuous authentication failure events in the hostd logs.
• Repeated "Cannot login user [email protected]: no permission" errors occurring every 5 minutes.
• The user agent appears as "pyvmomi 8.0.2.0.1 internal Python/3.11.11 (VMkernel; 8.0.3; x86_64)".
• These authentication failures generate thousands of log entries over time.
• This makes log analysis for troubleshooting other issues significantly more difficult.
• The failures occur despite successful NSX 4.2.x upgrades that resolved similar authentication issues in non-lockdown environments.

Environment

• ESXi hosts with lockdown mode enabled
• Occurs even with updated NSX-T 4.2.x with ESXi 8.0 Update 3e and newer hosts

Cause

The NSX Distributed Firewall (DFW) monitoring script attempts to authenticate as [email protected] every 5 minutes. The script uses pyVmomi API calls to collect CPU and memory statistics. ESXi lockdown mode security restrictions block direct root account authentication. This causes the script's authentication requests to fail with "no permission" errors while the monitoring process continues to retry at regular intervals.

Resolution

This issue is under review with Broadcom Engineering. Subscribe to the article to receive updates on this issue.

Workaround

Choose one of the following approaches based on your security requirements:

Option 1: Add root to lockdown mode exception list

This option requires security team evaluation:

1. Log in to the vSphere Client.
2. Navigate to the affected ESXi host.
3. Click Configure > System > Security Profile.
4. In the Lockdown Mode panel, click Edit.
5. Click Exception Users and add root to the list.
6. Click OK to apply the changes.
7. Monitor hostd logs to verify authentication failures are resolved:

   grep "Cannot login user [email protected]" /var/run/log/hostd.log
   

Option 2: Monitor until engineering solution is available

This is the recommended approach:

1. Verify current authentication failure frequency is manageable (every 5 minutes):

   grep "Cannot login user [email protected]" /var/run/log/hostd.log | tail -10
   

2. Implement log filtering for troubleshooting other issues to exclude these authentication events.
3. Subscribe to this article for updates on the engineering solution.

Additional Information

For more information about NSX authentication events that were resolved in NSX 4.2, see VCSA events show UserloginSessionEvent and UserLogoutSessionEvent for user [email protected].

For more information about ESXi lockdown mode configuration and exception users, see Enabling or disabling Lockdown mode on an ESXi host.

For more information about ESXi lockdown mode exception users, see Specifying Accounts with Access Privileges in Lockdown Mode.

To verify NSX component versions, run the following command on the affected ESXi host:

esxcli software vib list | grep nsx-exporter

To monitor authentication patterns in the hostd logs, use:

grep "[email protected]" /var/run/log/hostd.log | tail -20