"Encountered an internal error Traceback" "vmafd-firstboot error" while upgrading Windows vCenter Server from 6.7 U1 to 6.7 U3

Products

• VMware vCenter Server

Issue/Introduction

This article is to identify and resolve the "vmafd-firstboot" failure with backtrace complaining about "vecs_force_refresh".

Symptoms:

• Upgrading windows vCenter Server from 6.7 U1 to 6.7U3 fails at "vmafd-firstboot" with backtrace as shown below:

 

• You will see similar log entries in vminst.log file.

vminst.log
Traceback (most recent call last):
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 177, in main
    controller.firstboot()
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 55, in firstboot
    self.post_init()
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 63, in post_init
    service.post_init()
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdUpgrade.py", line 255, in post_init
    self.vecs_force_refresh()
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdInstall.py", line 672, in vecs_force_refresh
    (rc, stdout, stderr) = self.utils.run_command_with_display(command, cmddisplay)
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\IdentityUtils.py", line 112, in run_command_with_display
    stderr=subprocess.PIPE, stdin=subprocess.PIPE)
  File "C:\Program Files\VMware\vCenter Server\python\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Program Files\VMware\vCenter Server\python\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.7.x

Cause

This issue occurs when the 6.7 installer fails to remove all the vmware files, during the un-installation phase of 6.5 components as part of the upgrade process.

In above case this "C:\Program Files\VMware\vCenter Server\vmafdd\" directory had a 0 KB "vecs-cli" file, during un-installation of 6.5 components, the above file didn't get removed, post which the 6.7 installation installed all files, however during the firstboot of vmafd the "0 KB vecs-cli" file was being called as part of upgrade process, hence the "vmafd-firstboot" failed.

Resolution

To resolve this issue please follow the below steps.

1. Take a snapshot of the vCenter Server.
2. Go to the location C:\Program Files\VMware\vCenter Server\vmafdd\ and check if you can find the vecs-cli file.
3. If the file is 0 KB then remove the "0 KB vecs-cli" file.
4. Re-try upgrade process.

Additional Information

Impact/Risks:
No Impacts. 

Note: Please take a snapshot of "Windows vCenter" before initiating the upgrade.