Certificate management for VCF Instances unavailable in Fleet management due to configured Microsoft CA not reachable from SDDC Manager
search cancel

Certificate management for VCF Instances unavailable in Fleet management due to configured Microsoft CA not reachable from SDDC Manager

book

Article ID: 425493

calendar_today

Updated On:

Products

VCF Operations VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • The Certificate page appears empty in the Fleet Management UI.



  • When logging into SDDC Manager, the certificates are loading in the UI but taking a significantly long time to appear.



  • VCF Operation Cloud Proxy logs report the following error:

    /log/adapters/ManagementAdapter/ManagementAdapter_##.log
    yyyy-mm-ddThh:mm:ss.zzzZ ERROR ManagementAdapter 2414000 [ops@4413 threadId="3446" threadName="Collector worker thread 21" instanceId="98"] [(98) com.vmware.adapter.management.components.certificate.collect
    or.VCFCertificateCollector.lambda$fetchCertificatesForEachDomain$3] - Exception occurred in collecting certificates from domain - <domain_id>
    org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://<SDDC_FQDN>/v1/domains/<domain_id>/resource-certificates": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
            at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:791) ~[spring-web-5.3.43.jar:5.3.43]
            at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717) ~[spring-web-5.3.43.jar:5.3.43]
            at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:340) ~[spring-web-5.3.43.jar:5.3.43]
            at com.vmware.adapter.management.components.certificate.collector.VCFCertificateCollector.lambda$fetchCertificatesForEachDomain$3(Unknown Source) ~[vim.jar:?]
            at java.util.ArrayList.forEach(Unknown Source) ~[?:?]
    ..
    Caused by: java.net.SocketTimeoutException: Read timed out


  • SDDC Manager logs report the following error: 

    /var/log/vmware/vcf/operationsmanager/operationsmanager.log
    yyyy-mm-ddThh:mm:ss.zzz+0000 INFO  [vcf_om,0000000000000000,0000] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,om-scheduler-1] Fetching ca root certificates
    yyyy-mm-ddThh:mm:ss.zzz+0000 INFO  [vcf_om,0000000000000000,0000] [c.v.v.c.ca.plugin.MicrosoftCaService,om-scheduler-1] URI to fetch Microsoft CA certificate chain: https://<MSCA_SERVER_FQDN>/certsrv/certnew.p7b?ReqID=CACert&Enc=b64
    ..
    yyyy-mm-ddThh:mm:ss.zzz+0000 ERROR [vcf_om,0000000000000000,0000] [c.v.v.c.ca.plugin.MicrosoftCaService,om-scheduler-1] Unable to retrieve certificate from Microsoft CA
    yyyy-mm-ddThh:mm:ss.zzz+0000 ERROR [vcf_om,0000000000000000,0000] [c.v.v.c.ca.plugin.MicrosoftCaPlugin,om-scheduler-1] Unable to fetch Microsoft CA certificate
    com.vmware.vcf.certmgmt.common.exception.CertificateManagementException: org.apache.hc.client5.http.ConnectTimeoutException: Connect to https://<MSCA_SERVER_FQDN>:443 [<MSCA_SERVER_FQDN>/<MSCA_SERVER_IP>] failed: Connect timed out
            at com.vmware.vcf.certmgmt.ca.plugin.MicrosoftCaService.retrieveCertificate(MicrosoftCaService.java:351)
            at com.vmware.vcf.certmgmt.ca.plugin.MicrosoftCaService.getCaCertificate(MicrosoftCaService.java:563)
            at com.vmware.vcf.certmgmt.ca.plugin.MicrosoftCaPlugin.getCaCertificates(MicrosoftCaPlugin.java:102)
            at com.vmware.vcf.certmgmt.ca.plugin.MicrosoftCaPlugin.getCaCertificates(MicrosoftCaPlugin.java:115)
            at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.getRootCertificateAndInsertInMap(CertificateOperationOrchestratorImpl.java:1763)
            at com.vmware.vcf.certmgmt.service.orch.impl.CertificateOperationOrchestratorImpl.lambda$getCaRootCertificatesMap$66(CertificateOperationOrchestratorImpl.java:1751)
            at java.base/java.util.HashMap$KeySet.forEach(HashMap.java:1008)
    ..
    Caused by: java.lang.RuntimeException: org.apache.hc.client5.http.ConnectTimeoutException: Connect to https://<MSCA_SERVER_FQDN>:443 [<MSCA_SERVER_FQDN>/<MSCA_SERVER_IP>] failed: Connect timed out
            at com.vmware.vcf.secure.http.HttpClientService.execute(HttpClientService.java:1160)
            at com.vmware.vcf.certmgmt.ca.plugin.MicrosoftCaService.retrieveCertificate(MicrosoftCaService.java:336)
            ... 46 common frames omitted
    Caused by: org.apache.hc.client5.http.ConnectTimeoutException: Connect to https://<MSCA_SERVER_FQDN>:443 [<MSCA_SERVER_FQDN>/<MSCA_SERVER_IP>] failed: Connect timed out

Environment

VMware Cloud Foundation 9.x

Cause

The configured Microsoft CA is unreachable from SDDC Manager causing the certificate list retrieval API to timeout in VCF operations.

Resolution

Verify network connectivity between SDDC Manager and the configured Microsoft CA in TCP port 443.

Additional Information

SDDC Manager API / Certificates / Get Certificates By Domain

To troubleshoot the time it takes for the API to reply:

  1. Create a token on the SDDC Manager. (You will need to update the password value):

    TOKEN=$(curl -d '{"username" : [email protected], "password" : "updatepassword"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

  2. Retrieve the Domain ID (navigate to the Management Domain in the SDDC Manager UI and retrieve the value from the URL)
  3. Execute the API command to retrieve the certificates

    time curl -H "Authorization: Bearer $TOKEN" https://<SDDC-M_FQDN>/v1/domains/<Domain_ID>/resource-certificates --insecure

  4. Wait for the curl to provide an output and the amount of time it took to complete.
Powered by Wolken Software