Replacing Certificate for NSX-T via SDDC manager 3.9 gets stuck at pending status.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Replacing the Certificate for NSX-T  via SDDC manage 3.9  neither gets failed nor completes. Since certificate replacement does not complete,the whole workflow gets still at "IN-PROGRESS"

Operation manager log :

2020-01-30T07:59:16.112+0000 DEBUG [0622f1fdba64687d,25c6] [c.v.v.c.r.a.c.ResponseBuilder,http-nio-127.0.0.1-7300-exec-8] Build workflow object using domainCertificateOperation: { "workflowId": "aa0c3bc2-0d46-43ce-ad89-935406f5f8be", "domainName": "TDC-Critical", "operationType": "REPLACE_CERTIFICATE", "operationStatus": "INPROGRESS", "resourceCertificateOperations": [ { "resource": { "hostName": "FQDN of host", "resourceType": "vcenter", "master": false }, "result": { "status": "SUCCESSFUL", "message": "New certificate replaced successfully for FQDN of host." }, "creationTimestamp": 1580366723653, "updateTimestamp": 1580367466749 }, { "resource": { "hostName": "FQDN of host", "resourceType": "nsxt_manager", "master": false }, "result": { "status": "PENDING" },

2020-01-30T06:45:12.708+0000 INFO [0fc9850fc53b4255,0da8] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-6] Upload of file TDC-Critical.tar.gz completed
2020-01-30T06:45:19.691+0000 INFO [75b8bff8afbb4a16,2fed] [c.v.v.c.r.a.c.CertificateManagementController,http-nio-127.0.0.1-7300-exec-5]
Initiating certificate replacement operation with requestBody:
{ "domainName": "TDC-Critical", "resources": [ { "hostName": "FQDN of host" },
 { "hostName": "FQDN of host" }, { "hostName": "FQDN of host" },
 { "hostName": "FQDN of host" }, { "hostName": "FQDN of host" },
 { "hostName": "FQDN of host" } ] }

2020-01-30T06:47:55.258+0000 INFO [75b8bff8afbb4a16,b188] [c.v.v.c.vc.VCenterCertificatePlugin,om-exec-11] Replace certificate for resource: Resource(hostName=FQDN of host, id=9c35c848-0a44-4f3e-86eb-4b77ea6975d4, vmName=zetdccritvcs-01, resourceType=vcenter, credentials=[com.vmware.vcf.resourcedataprovider.model.Cre dential@37bee3e2, com.vmware.vcf.resourcedataprovider.model.Credential@35663285], ipAddress=192.xx.xxx.xxx, sans=[FQDN of host], master=false, clusterName=null)

 2020-01-30T06:57:46.742+0000 DEBUG [75b8bff8afbb4a16,b188] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,om-exec-11] New certificate replaced: SUCCESSFUL
2020-01-30T06:57:46.742+0000 DEBUG [75b8bff8afbb4a16,b188] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,om-exec-11] Replace certificate status: SUCCESSFUL

• Above logs indicate that vCenter certificate replacement went fine,however there is no logs related to NSXT certificate replacement here,which is still in PENDING state
• Ideally certificate would have replaced in below order:

2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host
2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host
2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host
2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host
2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host
2020-01-30T06:45:22.996+0000 DEBUG [75b8bff8afbb4a16,2fed] [c.v.v.c.s.a.ResourceDataProviderAssembler,http-nio-127.0.0.1-7300-exec-5] resources: FQDN of host

 

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.9.x

Cause

This issue occurs due to duplicate certificate replacement.

Resolution

This issue is resolved in 3.9.1 version. Please use the below  workaround with steps for replacing the NSXT certificate in VCF 3.9.

Workaround:
Locate the NSX-T manager VMs in the vCenter webClient and take a snapshot of each VM,
Refer below document to establish a trusted connection between NSX-T and vCenter

• https://docs.vmware.com/en/VMware-Validated-Design/5.0.1/com.vmware.vvd.sddc-nsxt-domain-deploy.doc/GUID-A3B08724-9C77-46D7-AFB2-0774B63A4AB6.html

If the above re-trust does not work, Proceed with below steps :

1.    Take snapshot for NSXT VM from vCenter UI.

2.   Update SDDC Manager trust-stores with Root and intermediate certificate using KB article https://kb.vmware.com/s/article/67726(if any applicable ).

Note:  If  same CA is used to sign the other certificates in the SDDC Manager, it is not needed to import CA keys in the SDDC manager trust stores again & this step can be skipped.

 

3.    SSH  to SDDC -Manager VM  , Change to root user and execute below:

 

cd /opt/vmware/vcf/operationsmanager/certificates/nsxt_csr_ids/

4.    cat <nsxt_fqdn> and note down csrid and which is needed in further steps.

5.    Open https://<nsxt_fqdn>/ui in browser and navigate to System -->Certificates  --> CSRs.

6.    Click on above noted CSRID  and click on actions icon.

7.    Click on Import certificate for CSR and copy the certificate in below order:

   Machine certificate --> Intermediate certificate (if any) --> Root certificate and  Click on Add button.

8.    Navigate to  System --> Certificates and note down certificate id for corresponding CSRID.

9.    SSH  to SDDC Manager, Change to root user and run the below command to hit API call.

 

curl -v -k -X POST "https://<nsxt_fqdn>/api/v1/node/services/http?action=apply_certificate&certificate_id=<certificate_id>"  -u 'admin:<password>' -H 'content-type: application/xml'

• Make sure the output is showing correct server certificate(Subject name/CN etc) ,if needed hit the API again (Incase sever certificate  details are not correct).
• After successful API response, Please wait for 3-4 mins to proceed further.
• Repeat above steps for all NSX-T nodes except  VIP/cluster node.

10.   SSH to SDDC Manager VM  and run the below command to hit  API call for  VIP node.

 

curl -v -k -X POST "https://<nsxt_vip_fqdn>/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=<certificate_id>"  -u 'admin:<password>' -H 'content-type: application/xml'