NSX Manager upgrade fails at Install_OS phase with 'ValueError: not enough values to unpack'.

Products

• VMware NSX

Issue/Introduction

• During an NSX Manager upgrade, the process fails on one of the nodes at the Install_OS step with the following error:

• The /var/log/syslog on the failed NSX Manager node displays the following traceback:

NSX 38833 SYSTEM [nsx@ comp="nsx-manager nsx-policy-manager nsx-controller" subcomp="upgrade-bundle" level="ERROR"] not enough values to unpack (expected 8, got 7)#Traceback (most recent call last):#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line 202, in run#self.run_action()#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line 103, in wrapper#func(self, *args, **kwargs)#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line xxx, in run_action#self.start_os_install()#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line 871, in start_os_install#self._config_password(from_passwd, from_shadow, to_passwd,#File "/image/VMware-NSX-unified-appliance-4.2.1.3.0/src/task.py", line xxx, in _config_password#userid, _, _,_, _, _ = line.split(":")#ValueError: not enough values to unpack (expected 8, got 7) 

• Based on the above error verified the /etc/shadow file and /etc/passwd file whether both of the files have similar accounts/users present or not, for instance:

on failed NSX Manager node : 

root@test:/# less /etc/passwd | grep user <<</etc/passwd
user1:x:10:100:User1:/home/user1:/usr/sbin/nologin
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

root@test:/# less /etc/shadow | grep user <<</etc/shadow
user1:x:10:100:User1:/home/user1:/usr/sbin/nologin

• If any of the users are missing at either /etc/shadow and /etc/passwd file, then the upgrade may fail.
  For example, here at /etc/shadow does not have a line for 'user2'

Environment

VMware NSX

VMware NSX-T Data Center

Cause

A user account exists in /etc/passwd but is missing from /etc/shadow (or vice versa). This mismatch occurs due to the manual deletion of a user account or automated modifications made by VMware Aria Automation.

Resolution

• Identify the missing user entry:

  Compare the contents of /etc/passwd and /etc/shadow to find the mismatched user account.
  
  grep user /etc/passwd
grep user /etc/shadow

• Add the missing line for the user into the respective file. Ensure the formatting matches the existing users exactly. For example, to add user2 to /etc/shadow:

root@test:/# less /etc/shadow | grep user <<</etc/shadow
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

• Verify that both users are added:

root@test:/# less /etc/shadow | grep user <<</etc/shadow

user1:x:10:100:User1:/home/user1:/usr/sbin/nologin
user2:x:11:101:User2:/home/user2:/usr/sbin/nologin

• Re-initiate the upgrade process and upgrade should be completed successfully this time for the failed NSX manager node.