NSX-T Hosts upgrade is stuck in progress while copying NSX bits to host

Products

• VMware NSX

Issue/Introduction

Symptoms:

• The status of the upgrade task for Hosts upgrade is on "In Progress" but does not move.
• The current task is stuck on "Copying NSX bits to host".
• The listed ESXi host entered Maintenance Mode successfully.
• NSX-T VIBs on the ESXi host were not upgraded yet: esxcli software vib list | grep nsx

[root@esxi01:~] esxcli software vib list | grep nsx
nsx-adf                        2.4.0.0.0-6.5.12454404                  VMware  VMwareCertified   2019-05-01
[...]

• NSX-T Manager logs repeat that the copy task is at 1%: get log-file syslog follow

<182>1 2019-06-01T10:00:00.000Z nsxtmanager1 NSX 4443 SYSTEM [nsx@6876 comp="nsx-manager" subcomp="upgrade-coordinator"] Component: HOST, status: IN_PROGRESS, % complete: 1.0, details: Copying NSX bits to host 192.#.#.#, canSkip: false

• ESXi /var/log/hostd.log show that the creation of a user 'nsxuser' was rejected as the password was too weak:

2019-06-01T09:50:00.000Z verbose hostd[2100424] [Originator@6876 sub=Solo.Vmomi opID=89c087c1 user=root] Arg user:
--> (vim.host.LocalAccountManager.AccountSpecification) {
-->    id = "nsxuser",
-->    password = (not shown),
-->    description = <unset>
--> }
2019-06-01T09:50:00.000Z info hostd[2100424] [Originator@6876 sub=Solo.Vmomi opID=89c087c1 user=root] Throw vmodl.fault.SystemError
2019-06-01T09:50:00.000Z info hostd[2100424] [Originator@6876 sub=Solo.Vmomi opID=89c087c1 user=root] Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.host.LocalAccountManager.weakPassword",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "reason",
-->                value = "short"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = "Weak password: too short."
-->    msg = ""
--> }

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

During the upgrade of ESXi hosts, the NSX-T Manager needs to create a user in order to copy the VIB upgrade bundle. This user is 'nsxuser'. If the password policy hardcoded in the NSX-T Manager is weaker than the password policy currently applied on the ESXi host, the user creation will fail and the upgrade will not be able to proceed.

Resolution

This issue is resolved in Firestar 2.5.0.

Workaround:
As a workaround, the password policy on the ESXi host can be temporarily disabled or weakened in order to allow the creation of 'nsxuser'.
To retry the upgrade, the service can then be restarted on the NSX-T Manager: service install-upgrade restart