Unable to list Orphaned VMs in VMware Cloud Director using cell-management-tool
Article ID: 430853
Updated On:
Products
Issue/Introduction
-
When attempting to list orphaned VMs using the "./cell-management-tool find-orphan-vms --host VCD_IP -u vcadmin -vc-name vcenter1 -vc-password P@55w0rd --vc-user admin --output-file -" command in VMware Cloud Director, the operation fails with an unexpected execution exception.
The error message indicates:
Unexpected error during command execution: Exception(s) during execution (See log file.), Root Cause: Exception(s) during execution (See log file.)Refer to cell-management-tool.log for more details
- Error messages similar to the following appear in /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file indicating a SSL handshake from VCD to vCenter with the following error:
DEBUG | pool-1-thread-9 | FindOrphanVmsCommand | Exception while processing vdc |com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:37780 to esapcredvcr01.esaginfra.local/##.##.##.##:443 failed in 76 msat com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261)at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:356)at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:57)at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)at com.sun.proxy.$Proxy159.retrieveContent(Unknown Source)at com.vmware.vcloud.clienttoolkit.vc.VcConnection.<init>(VcConnection.java:73)at com.vmware.vcloud.clienttoolkit.vc.VcConnection.<init>(VcConnection.java:65)at com.vmware.vcloud.cell.management.vcdclient.FindOrphanVmsCommand.findOrphans(FindOrphanVmsCommand.java:616)at com.vmware.vcloud.cell.management.vcdclient.FindOrphanVmsCommand.lambda$processVdc$4(FindOrphanVmsCommand.java:533)at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)at com.vmware.vcloud.cell.management.vcdclient.FindOrphanVmsCommand.processVdc(FindOrphanVmsCommand.java:540)at com.vmware.vcloud.cell.management.vcdclient.FindOrphanVmsCommand.lambda$executeCommand$0(FindOrphanVmsCommand.java:302)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)at java.base/java.lang.Thread.run(Thread.java:829)Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:37780 to vCenter_FQDN/##.##.##.##:443 failed in 76 ms
Environment
VMware Cloud Director 10.6.x
Cause
This is a known issue in VMware Cloud Director where the cell-management-tool (CMT) intermittently encounters closed database sessions causing the command execution to fail.
Resolution
Broadcom is aware of this issue and a fix is planned for an upcoming release of VMware Cloud Director. Please subscribe to this KB by following the steps in the article:Subscribe an article for future updates on the resolution.
Workaround:
As a temporary workaround, use the following query to identify VMs that exist in an Org VDC resource pool but are not managed by VMware Cloud Director (equivalent to the find-orphan-vms CMT command):
a. Log in directly or by using an SSH client to the primary appliance console as root.
b. Connect to the Cloud Director database using the below command:
sudo -i -u postgres psql vcloud
c. Run the following query to list the orphaned VMs:
WITH vmi AS ( SELECT vm_inv.moref, vm_inv.vc_id, org_prov_vdc_rp_view.rp_name AS res_group_name, org_prov_vdc_rp_view.rp_moref AS res_group_moref FROM vm_inv JOIN org_prov_vdc_rp_view ON vm_inv.resource_pool_moref = org_prov_vdc_rp_view.rp_moref AND vm_inv.vc_id = org_prov_vdc_rp_view.vc_id WHERE vm_inv.is_deleted = false AND vm_inv.is_vrouter = false AND NOT EXISTS (SELECT 1 FROM shadow_vm sv WHERE sv.vc_id = vm_inv.vc_id AND sv.moref = vm_inv.moref) AND NOT EXISTS (SELECT 1 FROM multi_vc_vm mvv WHERE mvv.vc_id = vm_inv.vc_id AND mvv.moref = vm_inv.moref) AND NOT EXISTS (SELECT 1 FROM disk d WHERE d.vc_id = vm_inv.vc_id AND d.shell_vm_moref = vm_inv.moref))SELECT vm_inv.name, vmi.res_group_name, vm_inv.moref, vm_inv.vc_idFROM vm_invJOIN vmi ON vmi.moref = vm_inv.moref AND vmi.vc_id = vm_inv.vc_idWHERE NOT EXISTS ( SELECT 1 FROM vm WHERE vm.moref = vm_inv.moref AND vm.vc_id = vm_inv.vc_id);
Feedback
