Unable to create a Workload Domain due to BoM component versions being outside of BoM specific versions

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Trying to add a host with a slightly different build but of the same version and its failing at the validations as part of the process outlined in Prepare ESXi Hosts for VMware Cloud Foundation
• When trying to create a workload the option is greyed out and says 

  No images found for (component) with version (version) 
  Ex: No images found for NSX-T Manager with version 3.1.3.7.0-1938045

• Error in /var/log/vmware/vcf/domainmanger/domainmanager.log

  INFO [vcf_dm,d541######,8b##] [c.v.e.s.c.s.a.i.ImageManagementServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-2]
  Got ESX product version for new Workload Domain: 7.0.3-20328353
  ERROR [vcf_dm,d541######,8b##] [c.v.v.h.s.v.HostVersionValidator,http-nio-127.0.0.1-7200-exec-2]
  Host hostname version 7.0.3-20842708 is not a compliant ESXi version, compliant versions are [7.0.3-20328353]
   ERROR [vcf_dmd541######,8b##] [c.v.v.h.s.v.HostVersionValidator,http-nio-127.0.0.1-7200-exec-2]
  Host hostname version 7.0.3-20842708 is not a compliant ESXi version, compliant versions are [7.0.3-20328353]

Environment

VMware Cloud Foundation 9.x

Cause

The validations fail when we try to deviate from the build specified in the BOM for the corresponding version of VCF.

For example:
Validations fail as below:

{
  "id": "#######-###-###-###-##########",
  "description": "Validating Domain Creation Spec",
  "executionStatus": "COMPLETED",
  "resultStatus": "FAILED",
  "validationChecks": [
    {
      "severity": "ERROR",
      "resultStatus": "FAILED",
      "errorResponse": {
        "errorCode": "INCOMPATIBLE_HOST",
        "arguments": [
          "hostname",
          "7.0.3-20842708",
          "[7.0.3-20328353]"
        ],
        "message": "Host hostname version 7.0.3-20842708 is not a compliant ESXi version, compliant versions are [7.0.3-20328353]"
      }
    }

Resolution

Considering the impact with day N-related operations adhere to the BoM-specified versions for all BoM Components when deploying WLDs in ideal scenarios.

Workaround 1:

1. Verify the VC, NSX and ESXi versions being used in the environment for the new WLD creation match the BoM versions for the corresponding VCF version.
2. Take a non-memory snapshot of the SDDC Manager VM.
3. SSH to SDDC manager as vcf user and su to root
4. Edit the application-prod.properties file:

   vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

5. Append it with the corresponding lines with values of VC, NSX-T, ESXi versions being used for the new WLD creation:
   

   For Example:
   vcenter.install.version=7.0.3.01000-20395099 
   nsxt.install.version=3.2.1.2.0-20541212 
   esx.install.version=7.0.3-20842708

   Note: Ensure that the versions used above are the ones to be used for the VI WLD Creation.
6. Restart lcm and verify that the service is running:

   systemctl restart lcm 
   systemctl status lcm

7. Re-run the validation and ensure is completes successfully.
8. Initiate the WLD creation

Workaround 2 

1. Take a non-memory snapshot of the SDDC Manager VM 
2. Remove *.install.version from lcm application-prod.properties
3. Restart LCM
4. If the install image is not available on the SDDC , Download the image