New Workload Domain deployment in a vXrail environment fails at the 'NSX-T Generate Input data' stage
search cancel

New Workload Domain deployment in a vXrail environment fails at the 'NSX-T Generate Input data' stage

book

Article ID: 384469

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

Details of the error seen in the SDDC UI:

Failed to fetch images from LCM Service with error.
Remediation Message: Download the necessary images and retry workflow

 

/var/log/vmware/vcf/domainmanager.log:


YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_dm,6756fba1542c13f05da077620dc31c17,3b16] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-16]  [2VMG75] FAILED_TO_FETCH_IMAGES Failed to fetch the images from LCM Service with error: .
com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Failed to fetch the images from LCM Service with error: .
        at com.vmware.evo.sddc.common.services.adapters.imagemanagementservice.ImageManagementServiceAdapterImpl.getDomainVersionMatrice
Caused by: com.vmware.cloud.foundation.rest.lcm.runtime.ApiException:
        at com.vmware.cloud.foundation.rest.lcm.runtime.ApiClient.handleResponse(ApiClient.java:788)
        at com.vmware.cloud.foundation.rest.lcm.runtime.ApiClient.execute(ApiClient.java:708)

 

/var/log/vmware/vcg/lcm/lcm-debug.log  indicates an 'InvalidStateException' when fetching the inventory:

 

YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_lcm,6756fb8065c95ba9132d3b2dc71a33bf,fbf1] [c.v.e.s.l.a.r.c.i.u.InventoryUpgradeContro
ller,http-nio-127.0.0.1-7400-exec-1] In InventoryUpgradeController, Exception in getting all inventory upgrades
com.vmware.evo.sddc.common.core.error.InvalidStateException: Management Domain not found

        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: MGMT domain collection size is not equal to 1
        at com.vmware.evo.sddc.lcm.services.impl.ImageManagementServiceImpl.getVersionComplianceMatrixForDomainInternal(ImageM
anagementServiceImpl.java:784)
        at com.vmware.evo.sddc.lcm.services.impl.ImageManagementServiceImpl.getVersionComplianceMatrixForDomain(ImageManagemen
tServiceImpl.java:334)
        ... 148 common frames omitted
Caused by: java.lang.RuntimeException: MGMT domain collection size is not equal to 1
        at com.vmware.evo.sddc.lcm.services.impl.ImageManagementServiceImpl.getVersionComplianceMatrixForDomainInternal(ImageM
anagementServiceImpl.java:477)
        ... 149 common frames omitted
YYYY-MM-DDTHH:MM:SS.###+0000 INFO  [vcf_lcm,6756fba1542c13f05da077620dc31c17,85a8] [c.v.e.s.l.a.r.c.i.ImageManagementControlle
r,http-nio-127.0.0.1-7400-exec-1] get version compliance matrix for domain VI
YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_lcm,6756fba1542c13f05da077620dc31c17,85a8] [c.v.e.s.l.s.i.ImageManagementServiceImpl,h
ttp-nio-127.0.0.1-7400-exec-1] MGMT domain collection size 0 is not equal to 1
YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_lcm,6756fba1542c13f05da077620dc31c17,85a8] [c.v.e.s.l.s.i.ImageManagementServiceImpl,h
ttp-nio-127.0.0.1-7400-exec-1] Error while getting version compliance matrix for VI domain - {}
java.lang.RuntimeException: MGMT domain collection size is not equal to 1

 

The workflow fails AFTER the vXrail cluster creation/first run while fetching vLCM images:

 

YYYY-MM-DDTHH:MM:SS.###+0000 INFO  [vcf_dm,675053b2e14ba4f21af975dc8888aec0,496d] [c.v.v.v.c.f.a.WaitForVxRailFirstRunToComplete,dm-exec-15]  VxRail first run workflow is completed successfully
2024-12-04T14:26:07.325+0000 DEBUG [vcf_dm,675053b2e14ba4f21af975dc8888aec0,496d] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-15]  Contract task Monitor VxRail First Run .

...............


YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_dm,675089425dfa2d772a50ac50568c58e6,886a] [c.v.e.s.c.s.a.i.ImageManagementServiceAdapterImpl,dm-exec-13]  Error in fetching Images from LCM

You may also encounter an issue where adding new hosts to an existing vXrail cluster, either by UI or API, fails.

The following error may be observed:

"Error in Fetching Workflow Options for Addition of ESXi Host to Cluster."

Logging similar to below may be observed:

domainmanager logging:

YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_dm,7c5fc00fcab54bcf,3c28] [c.v.v.n.c.v.v.ALBClusterValidator,http-nio-127.0.0.1-7200-exec-9]  Unable to get releases for domain 5a269a01-4e3c-4df6-b32c-358022e1ffb1. Status Code - 500, Response - {"errorCode":"VCF_ERROR_INTERNAL_SERVER_ERROR","arguments":[],"message":"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.","causes":[{"type":"com.vmware.evo.sddc.common.core.error.InternalServerErrorException","message":"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token."},{"type":"com.vmware.evo.sddc.lcm.model.error.LcmException"}],"referenceToken":"BHVOAV"}
YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_dm,d21826da4f4f439c,a534] [c.v.v.n.c.v.v.ALBClusterValidator,http-nio-127.0.0.1-7200-exec-9]  Unable to get releases for domain 5a269a01-4e3c-4df6-b32c-358022e1ffb1. Status Code - 500, Response - {"errorCode":"VCF_ERROR_INTERNAL_SERVER_ERROR","arguments":[],"message":"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token."
.........
        ... 146 common frames omitted
Caused by: com.vmware.evo.sddc.lcm.model.error.LcmException: null
        at com.vmware.vcf.lcm.rest.api.controller.v1.manifest.LcmReleaseController.getReleases(LcmReleaseController.java:173)
        ... 146 common frames omitted

lcm logging:

YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_dm,8d1caa4eb7ba419b,be09] [c.v.v.n.c.v.v.ALBClusterValidator,http-nio-127.0.0.1-7200-exec-2]  Unable to get releases
for domain b093c49f-8d24-4ec3-8a90-4aa773d71594. Status Code - 400, Response - {"errorCode":"DOMAIN_ID_INVALID","arguments":[],"message":"Provided domain ID is invalid.","referenceToken":"B0E1H8"}
com.vmware.vcf.rest.api.runtime.ApiException:
        at com.vmware.vcf.rest.api.runtime.ApiClient.handleResponse(ApiClient.java:788)
        at com.vmware.vcf.rest.api.runtime.ApiClient.execute(ApiClient.java:708)
        at com.vmware.vcf.rest.api.service.ReleasesApi.getReleasesWithHttpInfo(ReleasesApi.java:296)

lcm logs may also display the following entries:

YYYY-MM-DDTHH:MM:SS.###+0000 INFO  [vcf_lcm,1999b52c3575409a,75f6] [c.v.v.l.r.a.c.v.m.LcmReleaseController,http-nio-127.0.0.1-7400-exec-8] In LcmReleaseController, get release for domain b093c49f-8d24-4ec3-8a90-4aa773d71594
YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_lcm,1999b52c3575409a,75f6] [c.v.v.l.r.a.c.v.m.LcmReleaseController,http-nio-127.0.0.1-7400-exec-8] In LcmReleaseController, failed to get all releases
com.vmware.evo.sddc.lcm.model.error.LcmException: null
        at com.vmware.evo.sddc.lcm.adapter.inventory.InventoryClientHelper.getVersionForDomain(InventoryClientHelper.java:121)
        at com.vmware.evo.sddc.lcm.adapter.inventory.InventoryClientHelper.getVcfVersionForDomain(InventoryClientHelper.java:91)
        at com.vmware.evo.sddc.lcm.adapter.inventory.InventoryClientHelper.getVcfVersionForDomain(InventoryClientHelper.java:83)
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getVcfVersionForDomain(InventoryClientImpl.java:3272)
        at jdk.internal.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingRef

 

Node add never starts in the GUI, hangs on "fetching subtasks"

lcm logs may also display the following entries:

YYYY-MM-DDTHH:MM:SS.###+0000 ERROR [vcf_lcm,683775746b9cd1738386910101db551c,cab9] [c.v.e.s.e.h.VcfFallbackErrorController,http-nio-127.0.0.1-7400-exec-14] [HOPG8E] - Error attributes: {timestamp=Wed May DD H
H:MM:SS UTC YYYY, status=500, error=Internal Server Error, exception=java.lang.NullPointerException, trace=java.lang.NullPointerException: Cannot invoke "com.vmware.evo.sddc.lcm.model.internal.VCenter.getVCen
terId()" because "vcenter" is null
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getSsoCredentialFromDomain(InventoryClientImpl.java:1331)
        at jdk.internal.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)

Environment

VCF on vXrail 5.2.0 and 5.2.1

Cause

This issue is caused by a null value for the 'type' row in the vcenter table in the SDDC database for the new WLD vCenter:

Resolution

VMware is aware of this issue and are working toward a resolution.

Please engage Broadcom Support to assist in unblocking the WLD creation failure.

 

Powered by Wolken Software