vSAN ESA deployment fails during vSAN HCL validation in CloudBuilder

Products

• VMware Cloud Foundation

Issue/Introduction

• During vSAN ESA deployment with VCF Cloud Builder, the vSAN HCL validation workflow can encounter a failure.
• Specifically, the deployment process is blocked by the HCL check during the ESA validation phase.
• The VCF Cloud Builder performs a pre-validation, utilizing either an offline HCL JSON file or downloading a current version, to determine if the ESA drives are listed on the HCL.
• If the drives are not present, or if the validation process encounters any issues, the deployment is halted.

Environment

VMware Cloud Foundation

Resolution

Note: The workarounds should only be applied when vSAN ESA disks are on the HCL but the installer is not able to verify programmatically that they are supported, or should be applied only for POCs or non-production setups.

Workaround 1: Nested vSAN ESA Hardware Mock VIB

• The workaround is to circumvent vSAN ESA certified disk pre-check validations during VMware Cloud Foundation (VCF) deployments by injecting a hardware mock VIB into the ESXi installation
• Using this method, users can bypass these validations, facilitating deployments in environments where certified hardware may not be available. 

Steps:

1. Download the below pre-built VIB/Offline Bundle from the GIT repository:

nested-vsan-esa-mock-hw.vib (SHA256: f04d09b7b5392d53b9828f3c0b#######ad2e6d7fee29ca######2c0c4a10e)
nested-vsan-esa-mock-hw-offline-bundle.zip (SHA256: 04ec43a7307fd22edf893######0cc7bc0ef569b152a1e######b5e68b11b)

    2. Change ESXi acceptance level to CommunitySupported

        [root@esxi:~] esxcli software acceptance set --level CommunitySupported

    3.Install VIB

   [root@esxi:~] esxcli software vib install -v /nested-vsan-esa-mock-hw.vib --no-sig-check
   Installation Result
      Message: Operation finished successfully.
      VIBs Installed: xxxxxx.com_bootbank_nested-vsan-esa-mock-hw_1.0.0-1.0
      VIBs Removed:
      VIBs Skipped:
      Reboot Required: false
      DPU Results:

    4. Restart vSAN Management Service

        [root@esxi:~] /etc/init.d/vsanmgmtd restart

Note:  If the ESXi host has DPU, this approach will fail to install and has to be manually executed

 

Workaround 2: Python script to mock vSAN HCL 

Prerequisites: 

1. Enable SSH on all the ESXi hosts which failed during HCL validation.
2. Install prerequisites from a download of the requirements.txt file using below command if you want to proceed with third-party package option - This step is not required if you are not using a third party package:

         pip3 install -r requirements.txt

Steps to run the script:

     1. Download the hostSpectemplate.json and update the needed details.

     2. Prepare json as mentioned below for all failed hosts and copy to file.

[
    {
        "hostname": "hostname1.fqdn",
        "credentials": {
            "username": "username",
            "password": "password"
        }
    },
    {
        "hostname": "hostname1.fqdn",
        "credentials": {
            "username": "username",
            "password": "password"
        }
    }
]

    3. Download "mockVsanEsaHardware.py" on the cloud builder/vcf installer (recommended option) or any linux machine and run it:

    python3 <script_name> --hostSpec <json_file_path>

    4. You will be asked for an option, to either proceed with the native library(user intervention is needed for entering password) or use third party python packages(fully automated) (Step 2 in Prerequisites is required)

    5. Once execution of the script is complete, re-initiate the deployment workflow.

Note: 

• In the event of a host reboot, the script must be executed again. Mocking configurations are not persistent across reboots.
• The above mentioned workaround is recommended to be executed for POC purposes and not for the production environment.

Additional Information

Side effect:

Once you upload the hostSpectemplate.json on the ESXi hosts, all the controllers in the device list of the vCenter Server  UI will be shown as “mockedNVMeDevice”: