Reduced Downtime Upgrade (RDU) of vCenter Server fails with the error failed to set up service account 'vsphere-ui'

Products

• VMware vCenter Server

Issue/Introduction

Reduced Downtime Upgrade (RDU) from vCenter version 8.0 Update 3 and later to version 9.0 fails at the switchover phase, when starting the vsphere_ui services.

File based backup and restore(FBBR) was performed on source vCenter prior to the RDU upgrade. 

The /var/log/firstboot/vsphere-ui-prestart.log contains the below error,

 

YYYY-MM-DDTHH:MM:SS.194Z INFO vsphere-ui-prestart Found service account VSR file for vsphere-ui. Fetching service account values from the file.
YYYY-MM-DDTHH:MM:SS.223Z INFO vsphere-ui-prestart Found ndu marker file /var/cache/svcaccounts/ndu_marker_files/vsphere-ui
YYYY-MM-DDTHH:MM:SS.229Z ERROR vsphere-ui-prestart setup_vsphere_ui_svc_account_if_necessary - failed to set up service account 'vsphere-ui': [Errno 2] No such file or directory: '/var/cache/svcaccounts/vsphere-ui/.vsphere-ui'
YYYY-MM-DDTHH:MM:SS.229Z ERROR vsphere-ui-prestart Execution of vsphere-ui pre-start script failed!
Traceback (most recent call last):
File "/usr/lib/vmware-vsphere-ui/firstboot/vsphere_ui_prestart.py", line 220, in <module>
setup_vsphere_ui_svc_principal(VSPHERE_UI_FIRSTBOOT_CONFIG_DIR, \
File "/usr/lib/vmware-vsphere-ui/firstboot/solution_user_permission_utils.py", line 366, in setup_vsphere_ui_svc_principal
setup_vsphere_ui_svc_account_if_necessary(is_vmc_gateway)
File "/usr/lib/vmware-vsphere-ui/firstboot/solution_user_permission_utils.py", line 301, in setup_vsphere_ui_svc_account_if_necessary
setup_service_account(H5_CLIENT_SERVICE_ACCOUNT_NAME)
File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 1364, in setup_service_account
update_svc_password_file_owner_if_not_same(name, service_userid_name)
File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 922, in update_svc_password_file_owner_if_not_same
if service_userId is not None and not is_svc_password_file_owner_up_to_date(svc_name, service_userId):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 940, in is_svc_password_file_owner_up_to_date
if find_owner(svc_password_folder) == expected_owner and find_owner(svc_password_file) == expected_owner:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 952, in find_owner
return pwd.getpwuid(os.stat(path).st_uid).pw_name
^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/svcaccounts/vsphere-ui/.vsphere-ui'

 

 

Environment

vCenter 9.0

Cause

This issue occurs as the vsphere_ui service account password file goes missing after FBBR in vCenter 8.0U3 and above.

 

Resolution

To resolve the issue,

• Restore the system back to a healthy state. (Rollback snapshot/Restore from backup)
• Execute the attached script on the vCenter Server. This will fix the vsphere_ui service account. 
  
  

  #!/bin/python3
  
  import os
  import sys
  import logging
  
  sys.path.append(os.environ['VMWARE_PYTHON_PATH'])
  from cis.svcaccount_prestart_util import is_valid_account, setup_service_account
  
  from cis.utils import setupLogging
  setupLogging("fix_vsphere_ui_svcaccount", logMechanism="file")
  logger = logging.getLogger(__name__)
  logging.basicConfig(level=logging.CRITICAL)
  
  H5_CLIENT_SERVICE_ACCOUNT_NAME = 'vsphere-ui'
  setup_service_account(H5_CLIENT_SERVICE_ACCOUNT_NAME)
  
  if is_valid_account(H5_CLIENT_SERVICE_ACCOUNT_NAME):
      print("Successfully fixed the vsphere-ui service account")
  else:
      print("Failed to fix the vsphere-ui service account")

  
  

   

  The logs for this script will be present at /var/log/fix_vsphere_ui_svcaccount.log
  
  
• After executing this script, reattempt the RDU upgrade