"Internal error occurs during pre-upgrade checks"while upgrading to vCenter Server appliance 7.0

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Upgrading vCenter Server Appliance 6.7 to 7.0 fails with the error:

Internal error occurs during pre-upgrade checks.Resolution:
Send upgrade logs files to GSS for further assistance.

In the /var/log/vmware/upgrade/upgrade-requirements.log, you see message similar to:

2020-02-12T05:31:10.22Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log
Traceback (most recent call last):
File "bootstrap_scripts/run-preupgrade-checks.py", line 145, in main
commandFactory = getCommandFactory(upgradeParameters, execData, reporter)
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 4063, in getCommandFactory
factory = _getLinuxCommandFactory(upgradeParameters, settings, reporter)
File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 3960, in _getLinuxCommandFactory
upgradeParameters['source-host-thumbprint'])
File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py", line 1600, in createGuestOpsOperationManager
vmAddress, guestUser, guestPassword, sslThumbprint)
File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py", line 234, in __init__
.FindByIp(None, self._vmAddress, True)
File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 556, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 368, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1481, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'vim.fault.InvalidState',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'
}

2020-02-12T05:31:10.24Z INFO root Exiting with exit-code 1

In source hostd.log, you see message similar to:

2020-02-12T04:19:51.505Z info hostd[13947533] [Originator@6876 sub=Vimsvc.TaskManager opID=fb50909a user=root] Task Created : haTask--vim.SearchIndex.findByIp-54348
2020-02-12T04:19:51.514Z info hostd[13947442] [Originator@6876 sub=Default opID=fb50909a user=root] AdapterServer caught exception: N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState
--> )
--> [context]zKq7AVICAgAAAJA0nQATaG9zdGQAAAy3NWxpYnZtYWNvcmUuc28AAJDIGwDwuBcBg/ViaG9zdGQAAUQhagHSIWoB0hK1AQfBZAFg2GQBVeVkARbpZIJ2LAEBbGlidmltLXR5cGVzLnNvAAE1C8IBLRTCALVNKACTUSgAyyw2Azt9AGxpYnB0aHJlYWQuc28uMAAEHZsObGliYy5zby42AA==[/context]
2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Vimsvc.TaskManager opID=fb50909a user=root] Task Completed : haTask--vim.SearchIndex.findByIp-54348 Status error
2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Activation [N5Vmomi10ActivationE:0x000000836d711860] : Invoke done [findByIp] on [vim.SearchIndex:ha-searchindex]
2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg datacenter:
--> (null)
2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg ip:
--> "10.x.x.x"
2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg vmSearch:
--> true
2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Throw vim.fault.InvalidState
2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Result:
-->(vim.fault.InvalidState) {
-->faultCause=(vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""
--> }

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x

Cause

During upgrade prechecks, a call to the SearchIndex.findByIp VIM API might encounter an invalid VM during iteration. At this point, an error will occur and the call will fail before checking all VMs (usually this occurs when the invalid VM is traversed before the VM with the desired IP). This will fail upgrade prechecks.

Resolution

This is a known issue.

Currently, there is no resolution.

Workaround:
To workaround this issue:

Make sure there are no VMs on the host which would cause the call to fail. Specifically:

Make sure there are no VMs that are in the process of being created/deleted from disk/added to the inventory/reloaded - such VMs may also raise the same error so wait until these operations are completed.
Determine if there are any invalid VMs by running the command:

vim-cmd vmsvc/getallvms

Example: a 'Skipping invalid VM' warning will be listed for all invalid VMs

Skipping invalid VM '241'
Vmid Name File Guest OS Version

Examine the invalid VMs : VM's may be invalid due to issues with configuration, storage inaccessibility, missing disks/VMX files etc. If the issue cannot be remediated, make sure they do not interfere with the upgrade, unregister them from the host.

For example, using vim-cmd:

vim-cmd vmsvc/unregister 241

Execute the unregister command for each invalid VM.

Re-try the upgrade.