Putting ESXi hosts in lockdown mode disables the user nsx-user

Products

• VMware NSX

Issue/Introduction

Symptoms:
-- Host Preparation or transport node creation fails during new installation or an upgrade 
-- During a host VIB upgrade, the upgrade is stuck at "Copying NSX bits to host" and times out after several hours with Error "Permission Denied"
-- In the upgrade-coordinator.log, you see the following error a few hours after the upgrade commenced

2018-11-05 21:31:49.721 UTC INFO task-executor-0-workitem-HOST-6294251a-357f-4241-937b-2a6d398556c2 ExecutionMonitorServiceImpl:49 - SYSTEM [nsx@6876 comp="nsx-manager" subcomp="upgrade-coordinator"] Execution monitor service invoked to react to failure of node 6294251a-357f-4241-937b-2a6d398556c2 [No Permission to perform the operation on host https://10.252.25.15/sdk. Please check the Lockdown mode of the host.]

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

This issue occurs when lockdown mode is enabled on the ESXi host. When an ESXi host is put into lockdown mode, the user vpxuser is the only user who can authenticate with the host or run any commands. NSX-T Data Center relies on another user, nsx-user, to perform all NSX-T related tasks on the host.

Resolution

In a future release of NSX-T for data center, a pre-upgrade check for lockdown mode will be added during a host VIB installation. This check will fail for all hosts with lockdown mode enabled.

Workaround:
1. Disable lockdown mode
2. From NSX manager cli, run
              >restart service install-upgrade
3. Go back to the NSX manager UI and click retry/start to restart the upgrade

Additional Information

This is a known issue and addressed in the NSX-T 2.3 Release notes

Issue 1588682: Putting ESXi hosts in lockdown mode disables the user nsx-user