VCF 9.0.x Deployment using VCF Installer 9.0.x fails with error - 'NSX install image validation failed for NSX Version'
search cancel

VCF 9.0.x Deployment using VCF Installer 9.0.x fails with error - 'NSX install image validation failed for NSX Version'

book

Article ID: 414456

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware NSX

Issue/Introduction

  • VCF 9.0.x deployment using VCF installer 9.0.x failed at vSphere configuration stage.

  • VCF 9.0.x Installer returns error: "Unable to fetch Workflow Options" when hitting the DEPLOY button.



  • A similar error is seen on VCF Installer UI:



  • On the VCF 9.0.x. Installer, the workflow fails at the "Generate NSX Input Data" task with the following error:

    Product NSX_T_MANAGER install image not found for version 9.0.x.0.24952111.
Remediation: Ensure that the install image for product NSX_T_MANAGER with version 9.0.x.0.24952111 is downloaded and available

  • A similar error is found in /var/log/vmware/vcf/domainmanager/domainmanager.log in VCF Installer:
    Note: If the SDDC Manager and the Installer are different appliances, the below error message is seen on the SDDC Manager:

    DEBUG [vcf_dm,6######################9,b102] [c.v.e.s.o.d.OrchestratorDissemblers,http-nio-127.0.0.1-7200-exec-2]  The processing context being built is {"processingStateId":"ac10####-####-####-####-########0000","processingStateName":"VcfInstallerValidation_onFirstStart","refId":"VcfInstallerValidation_onFirstStart____0__","executionId":"86ba####-####-####-####-########5eb7","sddcId":"mgmt","status":"INITIALIZED",

     

  • A similar error is found in /var/log/vmware/vcf/domainmanager/domainmanager.log on VCF Installer:

    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,############] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-2]  Handler Error Response: {"errorCode":"WORKFLOW_OPTIONS_ERROR","arguments":[],"message":"Unable to fetch Workflow Options.","remediationMessage":"The requested operation could not be performed at the moment, please try again later. For details check logs.","causes":[{"type":"java.lang.IllegalArgumentException","message":"VMwareProductVersion can not be null or empty"}],"referenceToken":"######"}

     

  • A similar error is observed on the VCF Installer UI when the VCF Installer version differs from the target VCF Release 9.0.2

    "Get NSX Version Compliant with VMware Cloud Foundation"




Environment

VMware Cloud Foundation 9.0.x

VMware NSX 9.0.x

Cause

This is a known issue and is resolved in a future VCF Installer release.

Resolution

Use the VCF Installer 9.0.2 to deploy ALL VCF versions within the 9.0 release family.

VCF Installer Version Target VCF Release: 9.0.2 Target VCF Release: 9.0.1 Target VCF Release: 9.0.0
9.0.2 No additional steps required Download the 9.0.2 NSX and vCenter bundle + 9.0.1 bundles Download the 9.0.2 NSX-T and vCenter bundle + 9.0.0 bundles
9.0.1 Not Supported Run fix_manifest.sh with vcfManifestNew901.json on the VCF Installer Run fix_manifest.sh with vcfManifestNew900.json on the VCF Installer
9.0.0 Not Supported Not Supported Run fix_manifest.sh with vcfManifestNew900.json

Workaround #1

This workaround is effective only when applied to the VCF installer prior to initiating a new VCF deployment. If a deployment has already been started, apply Workaround #1 first, then proceed to Workaround #2.

Download both fix_manifest.sh and the corresponding manifest vcfManifestNew## file attached to this article. 

  1. Using a file transfer utility, upload both files to the VCF Installer appliance.

  2. SSH to the VCF Installer (or SDDC Manager depending on the scenario - If installer and SDDC are the same, login to the same appliance. If the installer and SDDC are separate appliances, apply the following steps on both the appliances).

  3. Elevate to root:
    su

  4. Navigate to the location of the uploaded files and update the file permissions:
    chmod +x fix_manifest.sh

  5. Run the following command:
    ./fix_manifest.sh '<admin-local-password>' vcfManifestNew###.json

Workaround #2

  1. SSH into the SDDC manager appliance with vcf user and then elevate to root with su.

  2. Identify the executionId UUID in the SDDC Manager domainmanager log, cat /var/log/vmware/vcf/domainmanager/domainmanager.log.

    Example:

    YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,#############,afa7] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7200-exec-3]  {"username":null,"timestamp":"YYYY-MM-DDTHH:MM:SS","clientIP":"127.0.0.1","userAgent":"Apache-HttpClient/5.3.1 (Java/17.0.16)","api":"/domainmanager/workflows/<executionId UUID>","httpMethod":"GET","httpStatus":200,"operation":"Get a workflow with a specific UUID","remoteIP":"127.0.0.1","duration":87}
    YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,#############,cebc] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7200-exec-8] {"username":null,"timestamp":"YYYY-MM-DDTHH:MM:SS","clientIP":"127.0.0.1","userAgent":"Apache-HttpClient/5.3.1 
    (Java/17.0.16)","api":"/domainmanager/workflows/<executionId UUID>","httpMethod":"GET","httpStatus":200,"operation":"Get a workflow with a specific UUID","remoteIP":"127.0.0.1","duration":77}
    YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,#############,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-8]  [#####] NSXT_INSTALL_IMAGE_VALIDATION_FAILED NSX install image validation failed for NSX Version - 9.0.2.0.25150386
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: NSX install image validation failed for NSX Version - 9.0.2.0.25150386
            at com.vmware.vcf.clustermanager.fsm.actions.ValidateNsxtInstallImage.execute(ValidateNsxtInstallImage.java:105)
            at com.vmware.vcf.clustermanager.fsm.actions.ValidateNsxtInstallImage.execute(ValidateNsxtInstallImage.java:38)
            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)      

  3. Enable domainmanager vault access and cycle the domainmanager service. 

    echo vcf.vault.http-access=true >> /etc/vmware/vcf/domainmanager/application.properties && systemctl restart domainmanager

  4. Run the following command to dump the workflow json.

    curl -X GET http://localhost/domainmanager/internal/vault/<executionId UUID identified in step 2> | json_pp > /tmp/workflow.json

    Sample:

    curl -X GET http://localhost/domainmanager/internal/vault/86ba####-####-####-####-########5eb7 | json_pp > /tmp/workflow.json

  5. Edit /tmp/workflow.json file and edit the line,"initSddcGreenfield____1__addViWithPolicyNsxt____4__NsxtAddViHeader____8__vcfCompliantNsxtVersion"

    vi /tmp/workflow.json

    Example for 9.0.1

    "initSddcGreenfield____1__addViWithPolicyNsxt____4__NsxtAddViHeader____8__vcfCompliantNsxtVersion" : "9.0.1.0.24952111",

  6. Update the modified workflow.json in VCF Installer

    curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/86ba####-####-####-####-########5eb7 -d @/tmp/workflow.json

  7. Retry the failed run from the VCF Installer UI

    POST SUCCESSFUL DEPLOYMENT

  8. SSH into the SDDC manager appliance with vcf user and then elevate to root with su.

  9. Truncate the manifest file to remove the custom manifest file. 

    psql -h localhost -U postgres -d lcm -c "truncate manifest"

Additional Information

In some instances, the depot may need to be disconnected to stop polling for a new manifest.

Attachments

vcfManifestNew901.json get_app
vcfManifestNew900.json get_app
fix_manifest.sh get_app
Powered by Wolken Software