"Internal error occurs during pre-upgrade checks" while upgrading to vCenter Server 7.0

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrading  vCenter Server to 7.0 fails with the error:
  Internal error occurs during pre-upgrade checks. Resolution:Send upgrade logs files to GSS for further assistance.

• In /var/log/vmware/upgrade/upgrade-requirements.log, there are messages similar to:

[YYYY-MM-DDTHH:MM:SS] ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log
Traceback (most recent call last):
File "bootstrap_scripts/run-preupgrade-checks.py", line 145, in main
commandFactory = getCommandFactory(upgradeParameters, execData, reporter)
...
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'vim.fault.InvalidState',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'
}

• In the source's hostd.log, there are message similar to:
  [YYYY-MM-DDTHH:MM:SS] info hostd[13947533] [Originator@6876 sub=Vimsvc.TaskManager opID=fb50909a user=root] Task Created : haTask--vim.SearchIndex.findByIp-54348
  2020-02-12T04:19:51.514Z info hostd[13947442] [Originator@6876 sub=Default opID=fb50909a user=root] AdapterServer caught exception: N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState
  ...
  [YYYY-MM-DDTHH:MM:SS] info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Throw vim.fault.InvalidState
  [YYYY-MM-DDTHH:MM:SS] info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Result:-->(vim.fault.InvalidState) {                                                                      -->faultCause=(vmodl.MethodFault) null,
  -->    faultMessage = <unset>
  -->    msg = ""
  --> }           

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x

Cause

During upgrade pre-checks, the call to the SearchIndex.findByIp VIM API may encounter an invalid VM during iteration. At this point, an error will occur and the call will fail before checking all VMs.

Resolution

Workaround:

Make sure there are no VMs on the host which would cause the call to fail. Specifically:

1. Make sure there are no VMs that are in the process of being created/deleted from disk, or added to the inventory. These VMs may need to wait until these operations are completed.

2. Determine if there are any invalid VMs by running the command:
$ vim-cmd vmsvc/getallvms

The 'Skipping invalid VM' warning will be listed for all invalid VMs 

Skipping invalid VM '241'
Vmid                    Name                                                             File                                                  Guest OS          Version    

3. Examine the invalid guest VMs. VM's may be invalid due to issues with configuration, storage inaccessibility, missing disks/VMX files, etc. If the issue cannot be remediated, unregister the problem VM from the host.

For example, using vim-cmd where the VMID is 241:
 

$ vim-cmd vmsvc/unregister 241

4. Re-try the upgrade.