When the SDDC Manager attempts to synchronize inventory versions with the VxRail Manager fails on error "Inventory version synchronization failed"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When viewing the task details or the Workload Domain status, will observe the following error message:
  

  Message: Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.
  Remediation Message: Please ensure the following: Please ensure <vxrail_FQDN> is running and reachable.
  Reference Token: #####
  Cause: Inventory version synchronization failed with: Unable to fetch VxRail Manager version for hostname <vxrail_FQDN> and user <username> to complete inventory sync.

• Following messages are seen in Operationmanager log file ,  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  

  YYYY-MM-DD DEBUG [vcf_om,698a8311931e3c3b6f0ae48b0e217d0e,f8ae] [c.v.v.s.t.DynamicTrustManager,om-exec-24] Error checking certificate chain CN=<vxrail_FQDN>, OU=<organization unit>, O=<organization>, L=<City>, ST=<state>, C=<country>
  sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
  at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
  YYYY-MM-DD WARN  [vcf_om,698a83c76e7c7953d70426c2c10e1dcf,2d07] [c.v.v.p.helper.GuestProgramService,om-exec-20] Could not find VMs with FQDN: <vxrail_FQDN>. (vmodl.fault.SystemError) {faultCause = null,
  YYYY-MM-DD ERROR [vcf_om,698aa05ded916fbb4a9d31e8c4af7a5c,c586] [c.v.e.s.o.model.error.ErrorFactory,pool-3-thread-5] [GS2RUT] VERSION_SYNC_FAILED Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.
  com.vmware.evo.sddc.common.core.error.InvalidStateException: Inventory version synchronization failed. Please try to fix the cause for the issue and retry the operation.

Cause

This issue is caused due to mismatch SSL/TLS trust relationship between the SDDC Manager and the VxRail Manager.

If the SSL certificate on the VxRail Manager appliance was recently renewed, regenerated, or replaced directly from the VxRail Manager interface (out-of-band), the SDDC Manager is unaware of this change and will reject the secure connection.

Resolution

To resolve this issue, add the newly generated VxRail Manager certificate to the SDDC Manager Trust Store for outbound communications.

1. Log in to the SDDC Manager UI.
2. Navigate to the left-hand menu and go to Inventory > Workload Domains.
3. Click on the name of the Workload Domain that contains the affected VxRail cluster.
4. Navigate to the Certificates tab.
5. Locate the VxRail Manager component in the list.  A warning or error in the Status column indicating an untrusted certificate is displayed.
   Note: A "Review" prompt directly within the error message banner at the top of the UI is seen.
   
6. Click Review and then choose to trust it.
7. Once the certificate is trusted, return to the Workload Domain summary or the failed task and Retry the inventory version synchronization. It should now complete successfully.