Cluster creation in SDDC manager fails with internal error due to NSX service account credentials issue in VCF 9.1
search cancel

Cluster creation in SDDC manager fails with internal error due to NSX service account credentials issue in VCF 9.1

book

Article ID: 456013

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • Deployment of a new cluster in the SDDC Manager fails with an "InternalServerError" during the Host Selection step.




  • /var/log/vmware/vcf/domainmanager/domainmanager.log on SDDC manager indicates an HTTP 403 error when fetching service account credentials for the NSX Manager.

    YYYY-MM-DDTHH:MM:SSZ DEBUG [vcf_dm,0000000000000000,0000] [c.v.v.c.f.p.n.s.s.CredentialMgmtServiceImpl,ForkJoinPool.commonPool-worker-1]  Fetching service account credentials for NSX 02###########################50
    YYYY-MM-DDTHH:MM:SSZ DEBUG [vcf_dm,0000000000000000,0000] [c.v.e.s.c.s.s.ServiceCredentialsHelper,ForkJoinPool.commonPool-worker-1]  Getting credentials for target type NSXT_MANAGER, entity ID 02###########################50 and service type SDDC_MANAGER
    YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_dm,0000000000000000,0000] [c.v.v.c.n.s.impl.NsxtWs1bServiceImpl,ForkJoinPool.commonPool-worker-1]  Failed to fetch the OIDC end point info for NSXmanager_FQDN {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}
    com.vmware.evo.sddc.common.core.error.FederatedBrokerIdentityException: {"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}
            at com.vmware.vcf.common.nsxt.services.impl.NsxtWs1bServiceImpl.lambda$errorHandler$1(NsxtWs1bServiceImpl.java:226)
            Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
    Error has been observed at the following site(s):
            *__checkpoint ⇢ Request to GET https://<NSXmanager_FQDN>/api/v1/trust-management/oidc-uris [DefaultWebClient]
    Original Stack Trace:
                    at com.vmware.vcf.common.nsxt.services.impl.NsxtWs1bServiceImpl.lambda$errorHandler$1(NsxtWs1bServiceImpl.java:226)
                    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
                    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
                    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
                    at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
                    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
                    at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2096)
                    at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145)
                    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
                    at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
                    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
                    at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
                    at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:455)
                    at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:509)
                    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:821)
                    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
                    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356)
                    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:434)
                    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
    YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_dm,0000000000000000,0000] [c.v.v.c.n.s.impl.NsxtWs1bServiceImpl,ForkJoinPool.commonPool-worker-1]   API response is null for :<NSXmanager_FQDN>

Environment

  • VMware Cloud Foundation (VCF) 9.1

Cause

The SDDC Manager database contains, locked, missing, or desynchronized credentials for the NSX service account, preventing successful API authentication with the NSX Manager.

Resolution

Powered by Wolken Software