NSX Manager Upgrade failed at Install_OS step.

Products

• VMware NSX

Issue/Introduction

• While performing the NSX manager Upgrade, upgrade process failed on 1 of the NSX managers with an error " MP upgrade failed at Install_OS step with error : An unexpected exception occurred: Value Error, not enough values to unpack (expected 8, got 7).

• From NSX Manager syslog (on failed node # /var/log/syslog) we could see the below error :

NSX 38833 SYSTEM [nsx@ comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="ERROR"] not enough values to unpack (expected 8, got 7)#Traceback (most recent call last):#File "/image/VMware-NSX-unified-appliance-4.2.1
.3.0/src/task.py", line 202, in run#self.run_action()#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line 103, in wrapper#func(self, *args, **kwargs)#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line xxx, in run_action#
self.start_os_install()#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line 871, in start_os_install#self._config_password(from_passwd, from_shadow, to_passwd,#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line xxx, in _config_password#
userid, _, _, _, _, _, _, _ = line.split(":")#ValueError: not enough values to unpack (expected 8, got 7) 

• Based on the above error verified the /etc/shadow file and /etc/passwd file whether both of the files have similar accounts/users present or not : for example below:

on failed NSX Manager node : 

root@test:/# less /etc/passwd | grep user <<</etc/passwd
user1:x:10:100:User1:/home/user1:/usr/sbin/nologin
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

 

root@test:/# less /etc/shadow | grep user <<</etc/shadow
user1:x:10:100:User1:/home/user1:/usr/sbin/nologin

• If any of the users are missing at either /etc/shadow and /etc/passwd , then the upgrade may fail. For example, here at /etc/shadow does not have a line for 'user2'

Environment

VMware NSX

VMware NSX-T Data Center

Cause

• This could be due to some manual deletion of the user or could be because of the VMware Aria Automation.

 

Resolution

• Add the missing line for the missing user. The format should be same as of the other user. At this example, we will just add below line at /etc/shadow:

root@test:/# less /etc/shadow | grep user <<</etc/shadow
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

• Verify that both users are added:

root@test:/# less /etc/shadow | grep user <<</etc/shadow

user1:x:10:100:User1:/home/user1:/usr/sbin/nologin
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

• Re-initiate the upgrade process and upgrade should be completed successfully this time for the failed NSX manager node.