Bringup fails during vCenter Server certificate replacement in VVD on VxRail

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• You see Configure Signed Certificate on vCenter Server task failed in cloud builder management console
• You see messages similar to the following in the bringup log file on the Cloud Builder VM:
  
  <---Invoking method POSTVALIDATE of task VcSignedCertificateConfiguration--->
  Description: Configure Signed Certificate on vCenter Server,
  ParamBuilder: param builder com.vmware.evo.sddc.bringup.adapters.toplugin.VcCertificateParamAdapter
  ProcessingId: ########-####-####-####-##########e7..
  2019-03-13T21:40:17.384+0000 DEBUG [0000000000000000,0000,operationToken=<########-####-####-####-##########71>] [c.v.e.s.b.m.c.SelfSignedBringupCertificates,threadPoolExecutor-4] Certificate will be generated for VC
  2019-03-13T21:40:17.384+0000 DEBUG [0000000000000000,0000,operationToken=<########-####-####-####-##########71>] [c.v.e.s.b.u.v.VcPscSignedCertificateConfiguration,threadPoolExecutor-4] Validating if thumbprint of certificate on host 10.249.236.11 matches 3E:4A:9F:96:6A:3F:7D:08:81:72:12:68:11:85:7A:D1:DD:B8:FB:91
  2019-03-13T21:40:17.397+0000 DEBUG [0000000000000000,0000,operationToken=<########-####-####-####-##########71>] [c.v.evo.sddc.bringup.util.HttpsUtil,threadPoolExecutor-4] SSL fingerprint for host 10.249.236.11 is: 80:46:6B:B0:3A:EE:1E:30:BA:D4:8F:03:3F:C4:31:B9:F0:BB:E1:3B
  2019-03-13T21:40:17.407+0000 ERROR [0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,threadPoolExecutor-4] [6NILN8] POSTVALIDATE_SIGNED_CERTIFICATE_CONFIGURATION_FAILED Failed to validate signed certificate configuration on 10.249.236.11
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to validate signed certificate configuration on 10.249.236.11
   at com.vmware.evo.sddc.bringup.plugin.vc.action65.impl.VcSignedCertificateConfiguration.postValidate(VcSignedCertificateConfiguration.java:124)
   at com.vmware.evo.sddc.bringup.plugin.vc.action65.impl.VcSignedCertificateConfiguration.postValidate(VcSignedCertificateConfiguration.java:30)
   at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$2(FsmActionState.java:22)
   at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
   at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:167)
   at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:154)
   at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:333)
   at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:460)
   at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:96)
   at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
   at reactor.bus.EventBus$3.accept(EventBus.java:317)
   at reactor.bus.EventBus$3.accept(EventBus.java:310)
   at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
   at reactor.bus.EventBus.accept(EventBus.java:591)
   at reactor.bus.EventBus.accept(EventBus.java:63)
   at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
   at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

 

• You see messages similar to the following in the /var/log/vmware/vpxd/vpxd.log file on the vCenter Server VM:

The remote host certificate has these problems:
-->
--> * unable to get issuer certificate)
--> [context]zKq7AVECAAAAAAnxsgANdnB4ZAAAiLEqbGlidm1hY29yZS5zbwAAQDcbAI67GACa+CEAZSkiABf9IQDPASIAXVIjAJUfIwBeIiMAEQkrAdRzAGxpYnB0aHJlYWQuc28uMAACnYwObGliYy5zby42AA==[/context]

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Validated Design for Software-Defined Data Center (SDDC) 5.0.x
VMware Validated Design for Software-Defined Data Center (SDDC)

Resolution

This is a known issue affecting VVD on VxRail. There is currently no resolution.

Workaround:
Use the following steps to workaround this issue:

1. Copy the correct RootCA.cer and IntermediateCA.cer files to the CertGenVVD root folder on the system where the CertGenVVD utility is installed.
2. Rename both files such that their filename extension is .pem.
3. Issue the following command:

openssl verify -verbose -CAfile RootCA.pem IntermediateCA.cer
IntermediateCA.pem: OK

 

Note: You may see an error similar to the following if the root or one of the intermediate certificates are invalid:

 

IntermediateCA.pem: C = US, O = DigiCert Inc, CN = DigiCert Global CA G2
error 20 at 0 depth lookup:unable to get local issuer certificate

4. Once the CA certificate chain is validated, concatenate the certificates together by issuing a command similar to the following:

copy IntermediateCAroot01.cer+IntermediateCAroot02.cer+RootCA.cer > Root64.cer

5. Use a file transfer utility to copy the Root64.cer file to the /tmp folder on the Cloud Builder VM.
6. SSH to the Cloud Builder VM as the root user and then issue the su - command to switch to the root user. 
7. Issue the following command to move the Root64.cer file from /tmp to /opt/vmware/vvd/certificates/RootCA/

mv /tmp/Root64.cer /opt/vmware/vvd/certificates/RootCA/

8. Issue the /opt/vmware/vvd/cloud-builder/install/reconfigure.sh command.
9. In the Cloud Builder UI, click the Retry button to restart the bringup process.