VMware Cloud Foundation upgrade fails with this error: ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS
search cancel

VMware Cloud Foundation upgrade fails with this error: ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS

book

Article ID: 344830

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
  • VMware Cloud Foundation upgrade fails with this error:

    ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS
  • In the /home/vrack/lcm/logs/lcm.log file, you see entries similar to:

    "errorDescription": "java.lang.RuntimeException: host is null\n\tat com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getEsxHostVIMConnection(InventoryClientImpl.java:986)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUpdateClientFactory.create(EsxUpdateClientFactory.java:49)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.runUpgrade(EsxPrimitiveImpl.java:198)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.startOrResumeUpgrade(EsxPrimitiveImpl.java:184)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.postUpgrade(EsxPrimitiveImpl.java:157)\n\tat com.vmware.evo.sddc.lcm.orch.PrimitiveService.postUpgrade(PrimitiveService.java:171)\n\tat com.vmware.evo.sddc.lcm.orch.PrimitiveService$$FastClassBySpringCGLIB$$aff213fb.invoke(\u003cgenerated\u003e)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)\n\tat org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)\n\tat java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.lang.Thread.run(Unknown Source)\n"

    Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment


Environment

VMware Cloud Foundation 2.1.x
VMware Cloud Foundation 2.0.x

Resolution

This is a known issue affecting VMware Cloud foundation. This issue is resolved in VMware Cloud Foundation 2.2.

To work around this issue:
 
Note: These steps must be completed from a Windows system with access to the 192.x.x.x network in the VMware Cloud Foundation environment.
  1. SSH to the VRM - VM as the root user.

    Note: The password for the root user can be obtained by running the / home/vrack/bin/vrm-cli.sh --lookup-password command on the VRM VM.
     
  2. Disable the firewall on the ISVM-1 VM by running this command:

    /opt/vmware/ism/scripts/ipsec/ipsecutil.py -c firewall-rules-delete
     
  3. Download and install the Enterprise version of DBeaver from http://dbeaver.jkiss.org/download/enterprise/ (installer + JRE). Accept all defaults during installation.
  4. Launch DBeaver. A Create new connection wizard will be displayed.
  5. Select Cassandra CQL as the connection type.
  6. Enter 192.168.100.43 as the Host. Leave the port as 9042.
  7. Enter vrmkeyspace as the Keyspace.
  8. Enter root as the User.
  9. Enter the root user's password as the Password.
  10. Click the Test Connection button. A dialog will be displayed stating whether the test was successful. Click OK. If the test was successful, click the Next button.
  11. Click the Next button on the Network screen.
  12. Enter a descriptive name in the Connection name field.
  13. Click the Finish button.
  14. Click the SQL Editor button.
  15. Run this query to obtain the hostname and IP address values related to the ESXi hosts in the VMware Cloud Foundation environment:

    select id,hostname,ipaddress,ipaddressmgmt from esxi

    Notes:
    • Press Ctrl-Enter to execute the statement.
    • Output similar to the following will be returned:

      id hostname ipaddress ipaddressmgmt
      5a5a7973-3ade-415e-a6bd-cbb11e8514fe rack-1-n3.vcf.corp.local 192.x.x.x
      510c5d00-a5a7-4b7e-8d00-6e79dc35ade2 rack-1-n5.vcf.corp.local 192.x.x.x
      4ff101a1-97ca-485d-84f6-c9a05257edcc rack-1-n0.vcf.corp.local 192.x.x.x 192.x.x.x
      2bece9e7-5002-4b87-9a1e-3bd173e0b91a rack-1-n2.vcf.corp.local 192.x.x.x 192.x.x.x
      1e2ace3e-9718-47c6-ad49-ea57f531ec24 rack-1-n4.vcf.corp.local 192.x.x.x
      2c3e27ec-81ae-4071-90b9-99ed1662cf8f rack-1-n1.vcf.corp.local 192.x.x.x 192.x.x.x

      Note: Identify the problematic ESXi host(s) by a missing ipaddressmgmt value. Only hosts that are in a workload domain should have an ipaddressmgmt value present. Any hosts that are not in a workload domain can be ignored. Make a note of the id value for all appropriate hosts.
  16. Add the appropriate management network IP address for the problematic host to the network table by executing:

    update esxi set ipaddressmgmt='<mgmt network IP>' where id=<id>

    Notes:
    • Press Ctrl-Enter to execute the statement.
    • Replace <mgmt network IP> with the appropriate management network IP address for the host in question. Replace <id> with the id value returned in Step 15 corresponding to the host in question.
       
  17. Verify that the host in question now has the correct management network IP address recorded by running this query:

    select id,hostname,ipaddress,ipaddressmgmt from esxi where id=<id>

    Notes:
    • Press Ctrl-Enter to execute the statement.
    • Replace <id> with the id value returned in Step 16 corresponding to the host in question.
    • You see output similar to:

      id hostname ipaddress ipaddressmgmt
      5a5a7973-3ade-415e-a6bd-cbb11e8514fe rack-1-n3.vcf.corp.local 192.x.x.x 192.x.x.x
       
  18. Repeat steps 16 and 17 for any other hosts identified in step 15.
  19. SSH to the ISVM-1 VM (192.168.100.43) as the root user.

    Note: The password for the root user can be obtained by running the /home/vrack/bin/vrm-cli.sh --lookup-password command on the VRM VM.
     
  20. Enable the firewall on the ISVM-1 VM by running this command:

    /opt/vmware/ism/scripts/ipsec/ipsecutil.py -c firewall-rules-create

    Note: The upgrade should now proceed in the VMware Cloud Foundation UI.


Powered by Wolken Software