Greenfield VCF deploy fails for "An error occurred in task 'Issue VMCA Certificate for NSX Managers'"
search cancel

Greenfield VCF deploy fails for "An error occurred in task 'Issue VMCA Certificate for NSX Managers'"

book

Article ID: 444766

calendar_today

Updated On:

Products

VCF Operations VMware vCenter Server VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

When performing a greenfield deployment of VMware Cloud Foundation (VCF), the installation fails during the NSX Manager certificate deployment phase. You will observe the following error message in the UI:

An error occurred in task "Issue VMCA Certificate for NSX Managers"

 

On the SDDC Manager in /var/log/vmware/vcf/domainmanager/domainmanager.log you messages similar to:

 ERROR [vcf_dm,6a31c3adc0be4a3b817be4a36a0d56d8,590d] [c.v.v.c.f.p.n.a.IssueVMCACertsForNsxtManagerAction,dm-exec-248]  UNABLE_TO_ISSUE_VMCA_SIGNED_CERTIFICATE
java.lang.RuntimeException: Retriable operation 'Issuing certificate to NSX manager: <nsx_manager_FQDN>' failed to complete after 3 retries.
 at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:221)
 at com.vmware.vcf.common.fsm.plugins.nsxt.action.IssueVMCACertsForNsxtManagerAction.issueCertificate(IssueVMCACertsForNsxtManagerAction.java:145)
 at com.vmware.vcf.common.fsm.plugins.nsxt.action.IssueVMCACertsForNsxtManagerAction.lambda$execute$0(IssueVMCACertsForNsxtManagerAction.java:100)
 at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
 at com.vmware.vcf.common.fsm.plugins.nsxt.action.IssueVMCACertsForNsxtManagerAction.execute(IssueVMCACertsForNsxtManagerAction.java:98)
 at com.vmware.vcf.common.fsm.plugins.nsxt.action.IssueVMCACertsForNsxtManagerAction.execute(IssueVMCACertsForNsxtManagerAction.java:37)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
 at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:401)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:515)
 at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
 at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
 at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
 at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) (statusCode:400) => {
    messages = [],
    data =  => {error_message=Certificate validation failed. Please use /api/v1/trust-management/certificates/<cert-id>?action=validate&usage=API API to check the validation errors., httpStatus=BAD_REQUEST, error_code=2078, module_name=internal-framework},
    errorType = INVALID_REQUEST
}
 at com.vmware.vapi.std.errors.InvalidRequest._newInstance2(InvalidRequest.java:195)
 at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
 at java.base/java.lang.reflect.Method.invoke(Method.java:580)
 at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.createStructBinding(JavaClassStructConverter.java:282)
 at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:86)
 at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:40)
 at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:349)
 at com.vmware.vapi.bindings.type.ErrorType.accept(ErrorType.java:37)
 at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:750)
 at com.vmware.vapi.internal.bindings.Stub.convert(Stub.java:549)
 at com.vmware.vapi.internal.bindings.Stub.convertError(Stub.java:566)
 at com.vmware.vapi.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:49)
 at com.vmware.vapi.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:28)
 at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultHttpResponseHandler.onResult(DefaultRequestExecutorFactory.java:94)
 at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:102)
 at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:52)
 at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:74)
 at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:288)
 at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:238)
 at com.vmware.nsx.trust_management.CertificatesStub.access$3000(CertificatesStub.java:20)
 at com.vmware.nsx.trust_management.CertificatesStub$Invoker.invoke(CertificatesStub.java:443)
 at com.vmware.nsx.trust_management.CertificatesStub$ApplyCertificateInvocationImpl.invoke(CertificatesStub.java:71)
 at com.vmware.nsx.trust_management.CertificatesStub$ApplyCertificateInvocationImpl.invoke(CertificatesStub.java:66)
 at com.vmware.vcf.common.nsxt.sdk.client.connection.ComplexHelpers.invoke(ComplexHelpers.java:169)
 at com.vmware.vcf.common.nsxt.sdk.client.connection.NodeCertOperations.applyCertificate(NodeCertOperations.java:56)
 at com.vmware.vcf.common.nsxt.sdk.client.connection.NsxtManagerCertOperations.applyCertificate(NsxtManagerCertOperations.java:43)
 at com.vmware.vcf.common.nsxt.services.impl.NsxtTrustManagementServiceImpl.applyCertificateToHttpServiceWithCustomTrust(NsxtTrustManagementServiceImpl.java:174)
 at com.vmware.vcf.common.nsxt.services.impl.NsxtTrustManagementServiceImpl.issueCASignedCertificateWithCustomTrust(NsxtTrustManagementServiceImpl.java:113)
 at com.vmware.vcf.common.fsm.plugins.nsxt.action.IssueVMCACertsForNsxtManagerAction.lambda$issueCertificate$1(IssueVMCACertsForNsxtManagerAction.java:133)
 at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:182)

 

Environment

  • VCF 9.1
  • VCF Installer 9.1
  • SDDC Manager 9.1
  • NSX Manager 9.1
  • vCenter Server 9.1

Cause

This issue occurs when the system time is out of sync by more than 30 seconds between the SDDC Manager, NSX Manager, and vCenter Server. If this time drift exceeds 30 seconds, the VMCA certificate cannot be successfully deployed to the NSX Manager, causing the VCF installation to fail. This is typically caused by an underlying issue with the configured NTP server failing to synchronize time correctly

Resolution

To resolve this issue, you must identify and fix the underlying Network Time Protocol (NTP) synchronization problem so that time is accurate across the environment.

  1. Log in to the SDDC Manager, NSX Manager, and vCenter Server via SSH.

  2. Run the date command on each appliance to verify the current system time and identify any time drift greater than 30 seconds.

  3. Check the status of the configured NTP servers by running the following command:
    ntpq -p

  4. Review the output. If the refid column displays . INIT. instead of a valid IP address, the appliance is failing to communicate or sync with that NTP server.

  5. Correct the network or configuration issue with the problematic NTP server so that time syncs correctly across all appliances.

  6. Once time is synchronized to within 30 seconds across the SDDC Manager, NSX Manager, and vCenter Server, retry the VCF deployment task.

Additional Information

Subscribe to this knowledge article to get updates on this issue.

Powered by Wolken Software