Root certificate expiration alarms in TRUSTED_ROOTS store
search cancel

Root certificate expiration alarms in TRUSTED_ROOTS store

book

Article ID: 326288

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Identify and manually remove expired or stale Certificate Authority (CA) certificates from the VMware Endpoint Certificate Store (VECS) TRUSTED_ROOTS store.
  • vCenter alarm for "Certificate Status" or "Root Certificate about to expire."
  • VMware Cloud Foundation (VCF) 9.x Brownfield import fails with an error indicating it could not retrieve trusted root certificates from vCenter.
  •  "VMware vSphere Profile-Driven Storage Service Health Alarm" and "vpxd service down" as potential symptoms of stale certificates.
  • VMware Analysis Tool (VDT) proactively flag these certificates during upgrade reviews.

Environment

  • VMware vCenter Server 7.x
  • VMware vCenter Server 8.x
  • VMware vCenter 9.x

Cause

The vCenter does not automatically purge expired or stale root certificates from the VECS TRUSTED_ROOTS store.

Resolution

Prerequisites:

  1. Perform a file-level backup via vCenter Server Appliance Management Interface (VAMI).
  2. Take a non-memory snapshot of the vCenter Server.
    Note: If the vCenter is in Enhanced Linked Mode (ELM) follow steps in KB: Creating a vCenter snapshot in Enhanced Linked Mode (ELM).
  1. List the certificates in the TRUSTED_ROOTS store to specifically list the Aliases and the Not After dates, by running the below command:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep "Alias\|Not After\|Subject:\|Issuer:"

  2. In the output for Step 1, note the Alias for the certificate that needs to be removed.

    Note:
    • There could be several certificates to remove. Any expired or not in use certificates should be removed to avoid certificate-related alarms. Look for Not After field in the certificate to identify the expiry date.

    • To view detailed list of all certificates in the TRUSTED_ROOTS store using vecs-cli run the command:
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less


     
  3. Verify that the root certificate intended to be removed is not currently in use by checking the "Authority Key Identifier" in the Machine SSL certificate against the "Subject Key Identifier" of the root certificate.

    1. Retrieve Machine SSL Certificate Details: Extract the Authority Key Identifier (AKI) from the current Machine SSL certificate:

      /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --text | grep -A 2 "Authority Key Identifier"

    2. Retrieve Root Certificate Details: Extract the Subject Key Identifier (SKI) from the suspicious certificate in TRUSTED_ROOTS:

      /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias [ALIAS_NAME] --text | grep -A 2 "Subject Key Identifier"

    3. Compare Identifiers:

      If the AKI and SKI values match: The certificate is currently in use. Do not remove it.
      If the values do not match: The certificate is safe to remove.

  4. Unpublish expired/expiring certificates from TRUSTED_ROOTS VECS Store:

    1. Using the Alias ID located in Step 2, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment:

      /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias ####################################2e7f --output /root/<aliasID>.cer

    2. Unpublish the expired/expiring CA certificate from VMDIR; it will prompt for SSO (Single Sign-On) Administrator credentials.

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer

    3. Delete the certificate from VECS utilizing the Alias located in step 4b:

      /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias ####################################2e7f

      Notes:
      If the alias has special characters, enclose the entire alias in single quotes when using the vecs-cli command, for eg. --alias 'https://[IP]:9997/vasa'
      If the command is failing with an error: "Operation failed with error ERROR_OBJECT_NOT_FOUND", ignore the error and proceed further. This error is received if the certificate is already removed from the store as part of step 4(b).

    4. Perform a force refresh of VECS to sync the certificate from VMDIR:

      /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

    5. Confirm that the certificate is no longer present.

      Note: Output of this command should not list the Alias ID that was removed in the above steps.

      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep <aliasID>

    6. Restart all services on the vCenter, ensure that all services start and respond normally, and that login and management of the environment are functioning properly.

      service-control --stop --all
      service-control --start --all

Additional Information

Powered by Wolken Software