SDDC Manager Precheck failure - Failed to obtain VIB list from ESXi host. Log into vCenter and ensure that the ESXi host is up and healthy

Products

• VMware Cloud Foundation

Issue/Introduction

• While running a precheck on a Workload Domain, it fails on one or more ESXi hosts.
  
• Running the command esxcli software vib list on the host successfully returns.
• No other pre-checks fail on the host.
• In logs /var/log/vmware/vcf/lcm/lcm-debug.log, entries similar to the below are found

  yyyy-mm-dd:hh:mm:ss ERROR [vef_lcm,xxxxxxxxxxxxxx,da32,precheckId=b7bc1277-3231-42d1-aba6-08975f675ff6,reaourceType=ESX, resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxcb37] [c.v.e.ckUtils, Precheck-24) Error occurred while performing third-party VIB installation check for Eax hoat with id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxcb37
  com. vmware.evo.sdde.lcm.model.error.LcmException: Failed to obtain VIB list from ESXi hoat. Log into vCenter and ensure that the ESXi host is up and healthy.
  		at com.waware.evo.adde.lem.primitive.impl.eax.EaxInternalModelTranslationservice.getViba(EaxInternalModelTranslationService.java:78)
  		at com.vaware.evo.adde.lem.primitive.impl.eax.EaxPrecheckUtils.precheckThirdPartyViba (EsxPrecheckUtils.java:275)
  		at com. wmware.evo.adde.lcm.primitive.impl.eax.EaxPrecheckUtils.doPrecheck(EaxPrecheckUtils.java:158)
  		at com. vaware.evo. adde.lcm.primitive.impl.esx.EaxVumPrimitiveImpl.precheck(EsxVumPrimitiveImpl.java:1020)
  		at com, wmware.evo. addc.lcm.services.impl.UpgradePrecheckAsyncMandler#1.run (UpgradePrecheckAsyncHandler.java:155)
  		at org. springframework.cloud. sleuth.instrument.async. TraceRunnable.run (TraceRunnable. java:64)
  		at java.base/java. util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
  		at java.base/java. util.concurrent.FutureTask.run (FutureTask. java:264)
  		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
  		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor. java:628)
  		at java.base/java.lang.Thread.run (Thread. java:829)
  Caused by: java.lang.NullPointerException: null
  		at com.vmware.vim. vmcmi.client.http.impl.HttpClient.resolveMoRef (HttpClient.java:124)
  		at com. vmware.vim. vmomi.client.http.impl.HttpClient.createStub(HttpClient.java:89)
  		at com. wmware.evo.adde.common.client.vmware.BaseVmomiClient.createStub (BaseVmomiClient.java:91)
  		as com.waware.evo.adde.common.client.vmware.eax.EaxCliClient.getEaxcliScapResult(EaxCliClient.java:106)
  		as com. vmware.evo.adde.lcm.primitive.impl.eax.EaxInternalModelTranalation3ervice.getVibs(EaxInternalModelTranalationService.java:63)
  		... 10 common frames omitted

Environment

VMware Cloud Foundation 3.7.x
VMware Cloud Foundation 3.8.x
VMware Cloud Foundation 4.x

Cause

This can be seen where there is a case sensitive discrepancy between the host FQDN in SDDC manager versus the DNS entry in the domain.

Resolution

1. Check the DNS resolution for the ESXi host(s) using nslookup <IP address of ESXi host(s)>
   • Ensure DNS record is not in Uppercase.
2. Checked host table in SDDC Platform database and ensure it is matching with the DNS record of the host
   1. ssh to sddc manager using vcf account and switch to root using su
   2. run the command 

      psql -h localhost -U postgres -d platform -c "select id,hostname,status from host where hostname='<Hostname_of_the_ESXi_host_which_has_a_precheck_failure>'"

      
      If any mismatches in the hostname then correct it by updating the hostname in DB
      1. Take snapshot of SDDC Manager VM
      2. SSH to SDDC Manager with VCF account and su to root
      3. run the command 

         psql -h localhost -U postgres -d platform -c "update host set hostname='<Correct_ESXi_hostname>' where id='<id_of_the_ESXi_host_found_in_step2(b)>'"

3. Check the DNS settings of the ESXi host under TCP/IP configuration and ensure domain name is configured
   • Navigate to vSphere Client > ESXi Host > Configure > TCP/IP Configuration > Default > DNS