Synchronize inventory versions fails in a VMware Cloud Foundation scale environment

Products

• VMware SDDC Manager

Issue/Introduction

Both inventory version synchronization and management domain upgrade precheck for out-of-band upgrade fail with the following cause:
"Inventory version synchronization failed with: Unable to fetch standalone ESX credentials via /v1/system/credentials API".

Environment

SDDC Manager 5.2.x

Cause

In large scale environments with unassigned hosts present in the VCF inventory, the version synchronization workflow and precheck for out-of-band upgrade timeout while trying to fetch host credentials.

Resolution

Failures can occur in several places:

• Upgrade precheck for management domain - Precheck WARNING for out-of-band upgrade can be ignored.
  
  
• Daily scheduled job - Failures of the daily sync job should not cause any issues and can be ignored.
  
  
• Version synchronization workflow - Sync workflow needs to be triggered per domain and not globally for the VCF inventory.
  Steps how to trigger version synchronization per domain can be found in this article https://knowledge.broadcom.com/external/article/324050/synchronize-inventory-versions-after-out.html. 
  Note: For unassigned hosts, version synchronization is blocked until SDDC Manager is upgraded to a version containing the resolution.