VCF 9.0.x Deployment using VCF Installer 9.0.x fails with error - 'NSX install image validation failed for NSX Version'
search cancel

VCF 9.0.x Deployment using VCF Installer 9.0.x fails with error - 'NSX install image validation failed for NSX Version'

book

Article ID: 414456

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware NSX

Issue/Introduction

  • VCF 9.0.x deployment using VCF installer 9.0.x failed at vSphere configuration stage.

  • VCF 9.0.x Installer returns error: "Unable to fetch Workflow Options" when hitting the DEPLOY button.



  • A similar error is seen on VCF Installer UI:



  • On the VCF 9.0.x. Installer, the workflow fails at the "Generate NSX Input Data" task with the following error:

    Product NSX_T_MANAGER install image not found for version 9.0.x.0.24952111.
    Remediation: Ensure that the install image for product NSX_T_MANAGER with version 9.0.x.0.24952111 is downloaded and available

  • A similar error is found in /var/log/vmware/vcf/domainmanager/domainmanager.log in VCF Installer:
    Note: If the SDDC Manager and the Installer are different appliances, the below error message is seen on the SDDC Manager:

    DEBUG [vcf_dm,6######################9,b102] [c.v.e.s.o.d.OrchestratorDissemblers,http-nio-127.0.0.1-7200-exec-2]  The processing context being built is {"processingStateId":"ac10####-####-####-####-########0000","processingStateName":"VcfInstallerValidation_onFirstStart","refId":"VcfInstallerValidation_onFirstStart____0__","executionId":"86ba####-####-####-####-########5eb7","sddcId":"mgmt","status":"INITIALIZED",

     

  • A similar error is found in /var/log/vmware/vcf/domainmanager/domainmanager.log on VCF Installer:

    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,############] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-2]  Handler Error Response: {"errorCode":"WORKFLOW_OPTIONS_ERROR","arguments":[],"message":"Unable to fetch Workflow Options.","remediationMessage":"The requested operation could not be performed at the moment, please try again later. For details check logs.","causes":[{"type":"java.lang.IllegalArgumentException","message":"VMwareProductVersion can not be null or empty"}],"referenceToken":"######"}

     

  • A similar error is observed on the VCF Installer UI when the VCF Installer version differs from the target VCF Release 9.0.2

    "Get NSX Version Compliant with VMware Cloud Foundation"




Environment

VMware Cloud Foundation 9.0.x

VMware NSX 9.0.x

Cause

This is a known issue and is resolved in a future VCF Installer release.

Resolution

Use the VCF Installer 9.0.2 to deploy ALL VCF versions within the 9.0 release family.

VCF Installer VersionTarget VCF Release: 9.0.2Target VCF Release: 9.0.1Target VCF Release: 9.0.0
9.0.2No additional steps requiredDownload the 9.0.2 NSX and vCenter bundle + 9.0.1 bundlesDownload the 9.0.2 NSX-T and vCenter bundle + 9.0.0 bundles
9.0.1Not SupportedRun fix_manifest.sh with vcfManifestNew901.json on the VCF InstallerRun fix_manifest.sh with vcfManifestNew900.json on the VCF Installer
9.0.0Not SupportedNot SupportedRun fix_manifest.sh with vcfManifestNew900.json

Workaround #1

This workaround is effective only when applied to the VCF installer prior to initiating a new VCF deployment. If a deployment has already been started, apply Workaround #1 first, then proceed to Workaround #2.

Download both fix_manifest.sh and the corresponding manifest vcfManifestNew## file attached to this article. 

  1. Using a file transfer utility, upload both files to the VCF Installer appliance.

  2. SSH to the VCF Installer (or SDDC Manager depending on the scenario - If installer and SDDC are the same, login to the same appliance. If the installer and SDDC are separate appliances, apply the following steps on both the appliances).

  3. Elevate to root:
    su

  4. Navigate to the location of the uploaded files and update the file permissions:
    chmod +x fix_manifest.sh

  5. Run the following command:
    ./fix_manifest.sh '<admin-local-password>' vcfManifestNew###.json

Workaround #2

  1. SSH into the SDDC manager appliance with vcf user and then elevate to root with su.

  2. Identify the executionId UUID in the SDDC Manager domainmanager log, cat /var/log/vmware/vcf/domainmanager/domainmanager.log.

    Example:

    YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,#############,afa7] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7200-exec-3]  {"username":null,"timestamp":"YYYY-MM-DDTHH:MM:SS","clientIP":"127.0.0.1","userAgent":"Apache-HttpClient/5.3.1 (Java/17.0.16)","api":"/domainmanager/workflows/<executionId UUID>","httpMethod":"GET","httpStatus":200,"operation":"Get a workflow with a specific UUID","remoteIP":"127.0.0.1","duration":87}
    YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,#############,cebc] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7200-exec-8] {"username":null,"timestamp":"YYYY-MM-DDTHH:MM:SS","clientIP":"127.0.0.1","userAgent":"Apache-HttpClient/5.3.1 
    (Java/17.0.16)","api":"/domainmanager/workflows/<executionId UUID>","httpMethod":"GET","httpStatus":200,"operation":"Get a workflow with a specific UUID","remoteIP":"127.0.0.1","duration":77}
    YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,#############,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-8]  [#####] NSXT_INSTALL_IMAGE_VALIDATION_FAILED NSX install image validation failed for NSX Version - 9.0.2.0.25150386
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: NSX install image validation failed for NSX Version - 9.0.2.0.25150386
            at com.vmware.vcf.clustermanager.fsm.actions.ValidateNsxtInstallImage.execute(ValidateNsxtInstallImage.java:105)
            at com.vmware.vcf.clustermanager.fsm.actions.ValidateNsxtInstallImage.execute(ValidateNsxtInstallImage.java:38)
            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)      

  3. Enable domainmanager vault access and cycle the domainmanager service. 

    echo vcf.vault.http-access=true >> /etc/vmware/vcf/domainmanager/application.properties && systemctl restart domainmanager

  4. Run the following command to dump the workflow json.

    curl -X GET http://localhost/domainmanager/internal/vault/<executionId UUID identified in step 2> | json_pp > /tmp/workflow.json

    Sample:

    curl -X GET http://localhost/domainmanager/internal/vault/86ba####-####-####-####-########5eb7 | json_pp > /tmp/workflow.json

  5. Edit /tmp/workflow.json file and edit the line,"initSddcGreenfield____1__addViWithPolicyNsxt____4__NsxtAddViHeader____8__vcfCompliantNsxtVersion"

    vi /tmp/workflow.json

    Example for 9.0.1

    "initSddcGreenfield____1__addViWithPolicyNsxt____4__NsxtAddViHeader____8__vcfCompliantNsxtVersion" : "9.0.1.0.24952111",

  6. Update the modified workflow.json in VCF Installer

    curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/86ba####-####-####-####-########5eb7 -d @/tmp/workflow.json

  7. Retry the failed run from the VCF Installer UI

    POST SUCCESSFUL DEPLOYMENT
  8. SSH into the SDDC manager appliance with vcf user and then elevate to root with su.

  9. Truncate the manifest file to remove the custom manifest file. 

    psql -h localhost -U postgres -d lcm -c "truncate manifest"

Additional Information

In some instances, the depot may need to be disconnected to stop polling for a new manifest.

 

If you are unsure if the SDDC Manager has been deployed on the same VM as the VCF Installer or as a Separate VM here is how to check -

 

This can be seen in VCF Installer log by searching for "Create SDDC with the following spec" and within that look for "sddcManagerSpec" element:

 

2026-04-20T13:21:48.440+0000 INFO  [vcf_dm,69e6286c7e72cee52aa2749fd58a1621,4c92] [c.v.v.v.c.v1.VcfInstallerController,http-nio-127.0.0.1-7200-exec-8]  Create SDDC with the following spec: { "sddcId": "SDDC ShortName", "workflowType": "VCF_EXTEND", "hostSpecs": [ { "hostname": "ESXi Host FQDN"....

-> "sddcManagerSpec": { "rootPassword": "*****", "hostname": "SDDC Manager FQDN",

You will then need to run Workaround 1 from above on the SDDC Manager VM also before retrying the task as you will pass the initial point of failure only for it to fail again on SDDC Manager pulling in incorrect 9.0.2 version

 

Attachments

vcfManifestNew901.json get_app
vcfManifestNew900.json get_app
fix_manifest.sh get_app