vCert - Scripted vCenter Expired Certificate Replacement
search cancel

vCert - Scripted vCenter Expired Certificate Replacement

book

Article ID: 385107

calendar_today

Updated On: 02-24-2025

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0 VMware vCenter Server

Issue/Introduction

vCert.py is a menu-driven tool that provides management capability for most certificate-related operations on the vCenter Server versions 7.0/8.0.

Environment

VMware vCenter Server 8.0

VMware vCenter Server 7.0 

Cause

The intent of the vCert script is to enable seamless replacement in the event that certificates in vCenter Server expire. 

Resolution

Table of Contents

Installation

The tool can be downloaded from this article and uploaded to vCenter Server. The script is executable by running the following command in the same directory the script is loaded:

# unzip -q vCert-6.0.0-20250218.zip
# cd vCert-6.0.0-20250218
# ./vCert.py

Running the Script

The following options are available in vCert.py

# ./vCert.py --help
usage: vCert.py [-h] [--version] [--env ENVIRONMENT] [--run OPERATION]
                [--user USER] [--password PASSWORD]
 
VCF Certificate Management Utility (version 6.0.0)
 
optional arguments:
  -h, --help           show this help message and exit
  --version            show program's version number and exit
  --env ENVIRONMENT    Config file for environment variables
  --run OPERATION      Run specific operation directly instead of showing menu
  --user USER          Specify an SSO administrator account
  --password PASSWORD  Password for the specified SSO administrator account


# ./vCert.py
 
------------------------!!! Attention !!!------------------------
 
This script is intended to be used at the direction of Broadcom Global Support.
 
Changes made could render this system inoperable. Please ensure you have a valid
VAMI-based backup or offline snapshots of ALL vCenter/PSC nodes in the SSO domain
before continuing. Please refer to the following Knowledge Base article:
https://knowledge.broadcom.com/external/article?legacyId=85662
 
Do you acknowledge the risks and wish to continue? [y/n]: y
 
 
VCF Certificate Management Utility (version 6.0.0)
-----------------------------------------------------------------
 1. Check current certificate status
 2. View certificate info
 3. Manage certificates
 4. Manage SSL trust anchors
 5. Check configurations
 6. Reset all certificates with VMCA-signed certificates
 7. ESXi certificate operations
 8. Restart services
 9. Generate certificate report
 E. Exit

The script will create a log file in the /var/log/vmware/vCert directory named vCert.log (which will be included in a support bundle), and will create a directory in /root/vCert-master with the name format YYYYMMDD, which will include several sub-directories for staging, backups, etc. Other than certificate backup files, the temporary files are deleted when the vCert tool exits.

 

Menu Options

Check current certificates status

Certificate health check for vCenter Server which includes the following:

  • If they are expired
  • If they will expire within 30 days
  • If the PNID is present in the Subject Alternative Name field (Machine SSL)
  • If the required Key Usage values are present (Machine SSL and Solution Users)
  • If a certificate contains any entries in the Subject Alternative Name field (Machine SSL and Solution Users)
  • If a certificate is configured to use the recommended Key Usage values (Machine SSL, Solution Users, STS signing certificate)
  • If a CA certificate has the Subject Key Identifier extension
  • If multiple CA certificates have the same Subject string (CA certs in VMware Directory & the TRUSTED_ROOTS store in VECS)
  • If a certificate is not a CA certificate (CA certs in VMware Directory, and the TRUSTED_ROOTS store in VECS)
  • If a certificate chain has a missing CA certificate
  • If a certificate has an invalid Alias (TRUSTED_ROOTS store in VECS)
  • If a certificate is using any of the unsupported signature algorithms (md2WithRSAEncryption, md5WithRSAEncryption, RSASSA-PSS, dsaWithSHA1, ecdsa_with_SHA1, and sha1WithRSAEncryption)
  • If the certificate for a Solution User in VECS matches the certificate for the Service Principal entry in VMware Directory
  • If a certificate is invalid because a signing CA violates the path length restrictions of a parent CA (Machine SSL, Solution Users, CA certs in VMware Directory & the TRUSTED_ROOTS store in VECS)

View Certificate Info

Prints human readable info for each of the following certificates:

View vCenter Certificates
-----------------------------------------------------------------
 1. Machine SSL certificate
 2. Solution User certificates
 3. CA certificates in VMware Directory
 4. CA certificates in VECS
 5. SMS certificates
 6. vCenter Extension thumbprints
 7. View STS signing certificates
 8. VMCA certificate
 9. Smart Card CA certificates
10. LDAPS Identity Source certificates
 R. Return to main menu

Options 1, 2, 8 will display the human-readable information and the fingerprint of the certificate(s). 
Options 3, 4, 5, 9, 10 will display a list of basic information (Subject, Issuer, validity dates, thumbprint, etc.), with the option to select from the list to view the human-readable information and the SHA1 fingerprint of the certificate.
Option 6 will display the configured thumbprint value for each of the vCenter extensions that should reference a thumbprint.
Option 7 will display a list of basic information (Subject, Issuer, validity dates, thumbprint, etc.).

Manage Certificates

Manage vCenter Certificates
-----------------------------------------------------------------
 1. Machine SSL certificate
 2. Solution User certificates
 3. CA certificates in VMware Directory
 4. CA certificates in VECS Directory
 5. SMS certificates
 6. vCenter Extension thumbprints
 7. STS signing certificates
 8. VMCA certificate
 9. Smart Card CA certificates
10. LDAPS Identity Source certificates
11. Clear expired certificates in BACKUP_STORE in VECS
12. Clear TRUSTED_ROOT_CRLS store in VECS
13. Clear Machine SSL CSR in VECS
 R. Return to main menu

Note: The script can accept DER or PEM Base64 encoded certificates, in standard, PKCS#7, or PKCS#12 format.

  1. This option replaces the Machine SSL certificate in VECS and updates the SSL trust anchors for the current node. A VMCA-signed certificate or custom CA-signed certificate can be used.
    • Custom CA-signed certificate - There is an option to generate a private key and Certificate Signing Request or import the signed certificate and key. If the presented CA-signed certificate does not include a complete CA chain then the script will prompt for a file containing the complete chain.
  2. This option replaces the Solution User certificates in VECS and updates the Service Principal entries in VMware Directory. The vpxd-extension thumbprints are updated in vCenter database. A VMCA-signed certificate or custom CA-signed certificate can be used.
    • Custom CA-signed certificates - There is an option to generate a private key and Certificate Signing Request or import the signed certificate and key. If the presented CA-signed certificate does not include a complete CA chain then the script will prompt for a file containing the complete chain.
  3. This option prints out the information of the CA certificates in VMware Directory, and another sub-menu with options to publish or remove certificates to VMware Directory.
  4. This option prints out the information of the CA certificates in VECS, and a prompt to delete one or more certificates (CA certificates should be published to VMware Directory in order to be added to the TRUSTED_ROOTS store in VECS).
  5. This option prints out the entries in the SMS store in VECS which includes the certificates used by the vmware-sps service, and the third-party VASA providers certificates that need to be trusted. The sub-menu has options to regenerate the SMS self-signed certificate and VMCA-signed certificate.
  6. Checks the vpx_ext table for mismatches on the thumbprints for the following extensions:
    • com.vmware.vim.eam (vpxd-extension)
    • com.vmware.rbd (vpxd-extension) [up to vCenter 8.0 U2]
    • com.vmware.vcIntegrity (vpxd-extension)
    • com.vmware.vlcm.client (vpxd-extension) [vCenter 8.0]
    • com.vmware.imagebuilder (vpxd-extension)
    • com.vmware.vsan.health (Machine SSL)
    • com.vmware.vmcam (Authentication Proxy)
    • If mismatches are detected, the user is prompted to update the extension thumbprints.
  7. This option backs up the current STS signing certificate and CA certificates, deletes all of the TenantCredential entries in VMware Directory, then creates a new entry with a VMCA-signed certificate.
  8. This option replaces the VMCA certificate and re-issues the Machine SSL, Solution User, and STS signing certificates. A VMCA-signed certificate or custom CA-signed certificate can be used. 
    • Custom CA-signed certificates - There is an option to generate a private key and Certificate Signing Request or import the signed certificate and key. If the presented CA-signed certificate does not include a complete CA chain then the script will prompt for a file containing the complete chain.
    • An option is available to replace the certificate with a self-signed signed certificate and to not regenerate certificates. This offers flexibility to regenerate only desired certificates after the machine SSL & VMSA certificate are replaced. 
  9. This option prints a list of certificates both in the reverse proxy Smart Card CA filter file, and the Smart Card issuing CA certificates in VMware Directory. The sub-menu has options to add/remove certificates to the filter file, and add/remove Smart Card issuing CA certificates to VMware Directory.
  10. This option prints a list of the certificates configured for AD over LDAPS Identity Sources (or ADFS in vCenter 7.0). The sub-menu has options to add or remove LDAP server certificates.
  11. This options clears the expired certificates in the BACKUP_STORE in VECS, and the BACKUP_STORE_H5C if it exists.
  12. This option clears the entries in the TRUSTED_ROOT_CRLS store in VECS.
  13. This option clears the __MACHINE_CSR entry in the MACHINE_SSL_CERT store in VECS. This is created when a Certificate Signing Request is generated from the vSphere Client, and contains the corresponding private key. 

Manage SSL Trust Anchors

Manage SSL Trust Anchors
-----------------------------------------------------------------
 1. Check SSL Trust Anchors
 2. Update SSL Trust Anchors

  1. This will output information for all unique certificates being used as SSL trust anchors for the Lookup Service registrations and the current Machine SSL certificates for all vCenter nodes in the SSO domain. The corresponding Service IDs and endpoint URIs can also be output, showing which certificate is being used for each service registration/endpoint.
  2. This option will update the SSL trust anchors for the selected vCenter Server within the SSO domain. This method modifies the entries directly rather than leveraging the Lookup Service scripts or libraries.

Check configurations

Configuration Check Menu
-----------------------------------------------------------------
 1. Check for SSL Interception
 2. Check STS server certificate and configuration
 3. Check VECS store status and permissions
 R. Return to main menu
 

  1. This option attempts to connect to hostupdate.vmware.com on port 443 and check the issuer of the presented certificate against "DigiCert TLS RSA SHA256 2020 CA1" to check for SSL Interception. If the certificate is issued by a different entity, the user is prompted to download the presented CA certificates and publish them to VMware Directory. If the CA chain is not complete, the user is notified that they will need to install the missing CA certificates manually. Once obtained, the missing CA certificates can be installed through this option.
  2. This option checks to see if the STS service is configured to use the certificate in the MACHINE_SSL_CERT store in VECS (used in greenfield 6.x/7.x deployments), or from the legacy STS_INTERNAL_SSL_CERT store (present if the system was upgraded from 5.5). If the STS service is configured for a store other than the MACHINE_SSL_CERT store, the user is prompted to update the configuration and restart the service.
    It will also check the vmwSTSConnectionStrings attribute from VMware Directory. On a standalone vCenter this value will reference the location of the VMware Directory instance by the vCenter hostname, but on a vCenter that is in Enhanced Linked Mode with other vCenter/PSC nodes in the SSO domain, this value should reference the VMware Directory instance via localhost . If this value is not set correctly, a prompt will be displayed to update it.
  3. This option checks the existence and permission assignments of the default stores in VECS. If any stores are missing, it will prompt to re-create them. If any expected permissions are not found on any of the stores, it will prompt to re-assign them.

Reset all certificates with VMCA-signed certificates

This option will reset the following certificates with ones signed by the VMCA:

Machine SSL (including SSL Trust Anchors and vCenter extension thumbprints)
Solution Users (including vCenter extension thumbprints)
STS Signing 

ESXi certificate operations

Manage ESXi Certificates
-----------------------------------------------------------------
 1. Check ESXi/vCenter certificate trust
 2. Check ESXi certificate against vCenter database
 3. Replace ESXi certificate

  1. Checks the TRUSTED_ROOTS store in VECS for the issuer of the certificates used by the reverse proxy (port 443) and the IOFilter VASA provider (port 9080) on the ESXi host. Checks for the issuer of vCenter's Machine SSL certificate, and the SMS certificate in the /etc/vmware/ssl/castore.pem file (the SMS certificate can also be present in the /etc/vmware/ssl/iofiltervp_castore.pem file).
  2. Compares the SSL certificate served by port 443 of the ESXi host(s) against the information in the vCenter database. If there is a discrepancy, this can cause issues with services like vSphere HA or vSAN on the hosts.
  3. Can replace the /etc/vmware/ssl/rui.crt, /etc/vmware/ssl/rui.key, and /etc/vmware/ssl/castore.pem file on the ESXi host.
    • The services on the host will need to be restarted for the new certificate and key to be applied.
    • The host will need to be Disconnected and Re-connected in vCenter to update the vCenter database with the new certificate information. 

Restart services

Restart VMware Services
-----------------------------------------------------------------
 1. Restart all VMware services
 2. Restart specific VMware service
 R. Return to main menu

  1. Restart all of the VMware services
  2. Enter a specific service to restart. This is checked against the list of services from the output of service-control --list

Generate certificate report

The generated report contains information the following:

  • All entries in VECS
  • All CA certificates in VMware Directory
  • All Service Principals (Solution Users) in VMware Directory
  • All STS signing certificate entries in VMware Directory
  • Various service certificates stored on the filesystem (Authentication Proxy, VMware Directory, etc.)
  • CA certificates for Smart Card Authentication
  • CA certificate for AD over LDAP(s) Identity Sources
  • SSL trust anchors for the Lookup Service registrations 

The report will be printed and will also be saved to a file in the /var/log/vmware/vCert directory.

Attachments

vCert-6.0.0-20250218.zip get_app
Powered by Wolken Software