VMware Cloud Foundation upgrade fails with this error: ESX_HOST_UPGRADE_STAGE_MAKE_VIM

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

VMware Cloud Foundation upgrade fails with this error:

ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS
In the /home/vrack/lcm/logs/lcm.log file, you see entries similar to:

"errorDescription": "java.lang.RuntimeException: host is null\n\tat com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getEsxHostVIMConnection(InventoryClientImpl.java:986)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUpdateClientFactory.create(EsxUpdateClientFactory.java:49)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.runUpgrade(EsxPrimitiveImpl.java:198)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.startOrResumeUpgrade(EsxPrimitiveImpl.java:184)\n\tat com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxPrimitiveImpl.postUpgrade(EsxPrimitiveImpl.java:157)\n\tat com.vmware.evo.sddc.lcm.orch.PrimitiveService.postUpgrade(PrimitiveService.java:171)\n\tat com.vmware.evo.sddc.lcm.orch.PrimitiveService$$FastClassBySpringCGLIB$$aff213fb.invoke(\u003cgenerated\u003e)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)\n\tat org.springframework.aop.interceptor.AsyncExecutionInterceptor$1.call(AsyncExecutionInterceptor.java:115)\n\tat java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.lang.Thread.run(Unknown Source)\n"

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment

Environment

VMware Cloud Foundation 2.1.x
VMware Cloud Foundation 2.0.x

Resolution

This is a known issue affecting VMware Cloud foundation. This issue is resolved in VMware Cloud Foundation 2.2.

To work around this issue:

Note: These steps must be completed from a Windows system with access to the 192.x.x.x network in the VMware Cloud Foundation environment.

SSH to the VRM - VM as the root user.

Note: The password for the root user can be obtained by running the / home/vrack/bin/vrm-cli.sh --lookup-password command on the VRM VM.
Disable the firewall on the ISVM-1 VM by running this command:

/opt/vmware/ism/scripts/ipsec/ipsecutil.py -c firewall-rules-delete
Download and install the Enterprise version of DBeaver from http://dbeaver.jkiss.org/download/enterprise/ (installer + JRE). Accept all defaults during installation.
Launch DBeaver. A Create new connection wizard will be displayed.
Select Cassandra CQL as the connection type.
Enter 192.168.100.43 as the Host. Leave the port as 9042.
Enter vrmkeyspace as the Keyspace.
Enter root as the User.
Enter the root user's password as the Password.
Click the Test Connection button. A dialog will be displayed stating whether the test was successful. Click OK. If the test was successful, click the Next button.
Click the Next button on the Network screen.
Enter a descriptive name in the Connection name field.
Click the Finish button.
Click the SQL Editor button.
Run this query to obtain the hostname and IP address values related to the ESXi hosts in the VMware Cloud Foundation environment:

select id,hostname,ipaddress,ipaddressmgmt from esxi

Notes:
- Press Ctrl-Enter to execute the statement.
- Output similar to the following will be returned:
  
  id hostname ipaddress ipaddressmgmt
  ########-####-####-####-########14fe rack-1-n3.vcf.corp.local 192.x.x.x
  ########-####-####-####-########ade2 rack-1-n5.vcf.corp.local 192.x.x.x
  ########-####-####-####-########edcc rack-1-n0.vcf.corp.local 192.x.x.x 192.x.x.x
  ########-####-####-####-########b91a rack-1-n2.vcf.corp.local 192.x.x.x 192.x.x.x
  ########-####-####-####-########ec24 rack-1-n4.vcf.corp.local 192.x.x.x
  ########-####-####-####-########cf8f rack-1-n1.vcf.corp.local 192.x.x.x 192.x.x.x
  
  Note: Identify the problematic ESXi host(s) by a missing ipaddressmgmt value. Only hosts that are in a workload domain should have an ipaddressmgmt value present. Any hosts that are not in a workload domain can be ignored. Make a note of the id value for all appropriate hosts.
Add the appropriate management network IP address for the problematic host to the network table by executing:

update esxi set ipaddressmgmt='<mgmt network IP>' where id=<id>

Notes:
- Press Ctrl-Enter to execute the statement.
- Replace <mgmt network IP> with the appropriate management network IP address for the host in question. Replace <id> with the id value returned in Step 15 corresponding to the host in question.
Verify that the host in question now has the correct management network IP address recorded by running this query:

select id,hostname,ipaddress,ipaddressmgmt from esxi where id=<id>

Notes:
- Press Ctrl-Enter to execute the statement.
- Replace <id> with the id value returned in Step 16 corresponding to the host in question.
- You see output similar to:
  
  id hostname ipaddress ipaddressmgmt
  ########-####-####-####-########14fe rack-1-n3.vcf.corp.local 192.x.x.x 192.x.x.x
Repeat steps 16 and 17 for any other hosts identified in step 15.
SSH to the ISVM-1 VM (192.168.100.43) as the root user.

Note: The password for the root user can be obtained by running the /home/vrack/bin/vrm-cli.sh --lookup-password command on the VRM VM.
Enable the firewall on the ISVM-1 VM by running this command:

/opt/vmware/ism/scripts/ipsec/ipsecutil.py -c firewall-rules-create

Note: The upgrade should now proceed in the VMware Cloud Foundation UI.

VMware Cloud Foundation upgrade fails with this error: ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS

Article ID: 344830

Updated On:

Products

Issue/Introduction

Environment

Resolution

Feedback