"com.vmware.vim.binding.vim.fault.SSLVerifyFault" Error during VM move, clone or vApp move, clone and template deployments

Products

• VMware Cloud Director

Issue/Introduction

• VM and vApp operations that happen across Provider VDCs backed by different vCenter Server instances throws an error "Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault" in VCD.
• In vCenter, the task fails with an error : Authenticity of the host's SSL certificate is not verified
  
  debug.log:
  2024-06-11 02:35:28,446 | ERROR    | vim-proxy-activity-pool-25 | RelocateVmActivity             | Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault | requestId=ab4bed23-69ce-463e-950d-XXXXXXXX,request=POST htt
  ps://VCD_URL/api/vApp/vapp-ce6ecfce-16fa-47d4-9814-XXXXXXXX/action/recomposeVApp,requestTime=1718073318939,remoteAddress=XXX.XXX.27.57:49358,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ..
  .,accept=application/*+xml;version 38.0.0-alpha vcd=XXXXXXXX-2d56-44e4-abb3-XXXXXXXX,task=94a2c112-ebd8-4ad1-abfc-XXXXXXXXactivity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:94a2c112-ebd8-4ad1-abfc-34
  bd12b797ba) activity=(com.vmware.vcloud.vdc.impl.ComposeRecomposeVAppActivity,urn:uuid:1b702e39-73a4-4a15-8f0e-XXXXXXXX) activity=(com.vmware.vcloud.vdc.impl.LinkVMsToTargetVAppActivity,urn:uuid:XXXXXXXX-b565-4fa6-a635-XXXXXXXX)
  activity=(com.vmware.vcloud.vdc.impl.MoveVmUsingVmotionActivity,urn:uuid:e2ec244a-bb4c-4450-bf89-XXXXXXXX) activity=(com.vmware.ssdc.backend.services.impl.RelocateVmActivity,urn:uuid:3909fefe-5fa7-4a70-a565-XXXXXXXX) activity=(co
  m.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByStorageClassActivity,urn:uuid:XXXXXXXX-1c4b-4df5-bb92-XXXXXXXX) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByDatastoreActivity,urn:uuid:fcf8
  49c7-ec04-47dd-8ec9-XXXXXXXX) activity=(com.vmware.vcloud.val.internal.impl.RelocateVmActivity,urn:uuid:d6d8c0ec-9f66-4931-9917-XXXXXXXX)
  com.vmware.ssdc.library.exceptions.GenericVimFaultException: Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault
          at com.vmware.ssdc.library.vim.LmVim.createGenericVimFaultException(LmVim.java:329)
          at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:445)
          at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
          at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity.getResultIfTaskAlreadyCompleted(AsynchronousTaskWaitActivity.java:449)
          at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$InitialPhase.invoke(AsynchronousTaskWaitActivity.java:123)
          at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
          at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
  Caused by: (vim.fault.SSLVerifyFault) {
     faultCause = null,
     faultMessage = null,
     selfSigned = false,
     thumbprint = XX:C1:XX:1C:XX:79:XX:8B:XX:AA:XX:56:XX:5C:XX:E5:XX:19:XX:B4

Environment

VMware Cloud Director 10.4

Cause

Starting with VMware Cloud Director 10.4.1 VM and vApp operations that happen across Provider VDCs backed by different vCenter Server instances require that both vCenter Servers trust each other's certificates.

Resolution

• If there are more than one vCenter Server managed by VMware Cloud Director, make sure that there is a mutual trust across all vCenter Server instances.
• If VMCA or user's internal Enterprise CA is used to issue certificates then it must be imported between vCenter server to enable trust.

Steps to trust certificates between vCenter servers:

• The source VMCA certificate can be retrieved via the download.zip from http://<source-VC>/certs/download.zip, extracted and uploaded via the ADD UI.
• vCenter Server provides UI for certificate management in vSphere Client (Administration > Certificates > Certificate Management).