ESXi prechecks are failing for SDDC 5.X, when planning an upgrade to vCF9 "Check if ESXi host boot device and boot bank size are sufficient"/ "Internal error occurred while trying to perform third party VIBs validation"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Running prechecks for all ESXi hosts may fail with the following errors:

Error 1.
Check if ESXi host boot device and boot bank size are sufficient
ERROR
ESXi storage system check failed on host with error: null
High: Do not perform upgrade without addressing this issue.
Please ensure connection credentials are up-to-date and there is network connectivity to the ESXi host

Error 2.
Internal error occurred while trying to perform third party VIBs validation
ERROR 
While trying to perform third party VIBs validation internal error occurred. Host has following third-party VIBs java.lang.NullPointerException
High: Do not perform upgrade without addressing this issue.
Host has following third-party VIBs java.lang.NullPointerException. For more details visit 'Third-Party Custom VIBs' page 'Update/Patches -> Third-Party Custom VIBs'

There are no boot device issues, no unsupported 3rd party VIBs 

Logs state the following
- Operations manager references FQDN (Log Location: cd /var/log/vmware/vcf/operation manager)

<Date && Time> DEBUG [vcf_om,6995ce4152e0df97fe07f3ac0e810347,006d] [c.v.e.s.c.c.v.vsphere.VcManagerBase,precheck-validation8] Connecting to https://<HOSTNAME_FQND>:443/sdk
[...]
<Date && Time> DEBUG [vcf_om,6995ce4152e0df97fe07f3ac0e810347,006d] [c.v.e.s.c.c.vmware.esx.EsxCliClient,precheck-validation8] Creating HostSystem stub for host with MOR: null
<Date && Time> DEBUG [vcf_om,6995ce4152e0df97fe07f3ac0e810347,006d] [c.v.e.s.c.c.v.vsphere.VsphereClient,precheck-validation8] Destroying 1 open views
<Date && Time> INFO  [vcf_om,6995ce4152e0df97fe07f3ac0e810347,006d] [o.b.jsse.provider.ProvTlsClient,precheck-validation8] [client #41925 @7a68eb0c] disconnected from <ESXI_FQDN>:443

•  However, the host is currently referenced only by its short hostname instead of the fully qualified domain name (FQDN).

less /var/run/log/hostd.log

<Date && Time> In(166) Hostd[2098713]: [Originator@6876 sub=Vimsvc.TaskManager opID=xxxxxx sid=xxxxxx user=<ESXI_SDDC -SVC Account>] Task Created : haTask--vim.SearchIndex.findByDnsName-xxxxx
<Date && Time> In(166) Hostd[2098727]: [Originator@6876 sub=Vimsvc.TaskManager opID=782ffd15 sid=523426d4 user=<ESXI_SDDC -SVC Account>] Task Completed : haTask--vim.SearchIndex.findByDnsName-19311272 Status success
[...]
<date && Time> In(166) Hostd[2098727]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=782ffd6a sid=52121aad] Event 135359 : User <ESXI_SDDC -SVC Account>@<SDDC IP> logged in as VMware vim-java 1.0
<date && Time> In(166) Hostd[2098737]: [Originator@6876 sub=Vimsvc.TaskManager opID=782ffd6e sid=52121aad user=svc-ESXI_fqdn] Task Created : haTask-ha-host-vim.host.ServiceSystem.start-19311299
<date && Time> In(166) Hostd[2098711]: [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
<date && Time> In(166) Hostd[2098701]: -->    key = 12,
<date && Time> In(166) Hostd[2098701]: -->    chainId = 0,
<date && Time> In(166) Hostd[2098701]: -->    createdTime = "<Date && TIME>",
<date && Time> In(166) Hostd[2098701]: -->    userName = "",
<date && Time> In(166) Hostd[2098701]: -->    host = (vim.event.HostEventArgument) {
<date && Time> In(166) Hostd[2098701]: -->       name = "<Host Shortname>",
<date && Time> In(166) Hostd[2098701]: -->       host = 'vim.HostSystem:ha-host'
<date && Time> In(166) Hostd[2098701]: -->    },
<date && Time> In(166) Hostd[2098701]: -->    eventTypeId = "esx.audit.ssh.enabled",
<date && Time> In(166) Hostd[2098701]: -->    objectId = "ha-host",
<date && Time> In(166) Hostd[2098701]: -->    objectType = "vim.HostSystem",

Environment

SDDC 5.X

VCF 9.0

VCF 9.0.1

Cause

The hostname retrieved via the esxcli command is returning the short hostname instead of the fully qualified domain name (FQDN).

# esxcli system hostname get
   

Domain Name:
Fully Qualified Domain Name: <Host Short name>
Host Name: <Host Short name>

Under normal circumstances, the precheck workflow functions as follows: the EsxiThirdPartyVIBSValidator expects the ESXi host FQDN from the EsxCliClient, after which the workflow proceeds to the VcManager and subsequent components.

However, since the environment is returning only the short hostname instead of the FQDN, the workflow receives a NULL value. As a result, the generated responses are false positives.

Resolution

Set the domain name on the ESXi host so that the DnsConfig.domainName is populated:

# esxcli system hostname set --domain=<domain.name>

# esxcli system hostname get

   Domain Name: <domain name>
   Fully Qualified Domain Name: <Host FQDN>
   Host Name: <Host Short name>

 

If the above doesn't work, follow the below steps:

• Navigate to the vCenter Server UI.

• Select the affected ESXi host > Configure > Networking > TCP/IP configuration.

• Select the Default TCP/IP stack > Edit.

• Configure the correct Domain name and Host name to establish the complete FQDN.

• Rerun the SDDC Manager precheck.

Additional Information

While a code fix is planned for SDDC in future releases, to accept short name as well, make sure that the nslookup/ DNS entry for the Hosts replies with the FQDN.