vCert reports a MISMATCH for extension thumbprints after a vCenter 8.0.3 to 9.1 upgrade

Products

• VMware vCenter Server

Issue/Introduction

vCert reports a MISMATCH for extension thumbprints after a vCenter 8.0.3 to 9.1 upgrade. This may include extensions com.vmware.vcIntegrity, com.vmware.vim.eam, com.vmware.vlcm.client, and com.vmware.vsan.health.

Checking vCenter Extension Thumbprints 
-----------------------------------------------------------------
 com.vmware.vcIntegrity (vpxd-extension) MISMATCH
 com.vmware.vim.eam (vpxd-extension) MISMATCH
 com.vmware.vlcm.client (vpxd-extension) MISMATCH
 com.vmware.vmcam (Authentication Proxy) MATCHES
 com.vmware.vsan.health (Machine SSL) MISMATCH

 

In the vCert log on vCenter, in /var/log/vmware/vCert/vCert.log, what you will see if extension thumbprints are actually empty as opposed to a MISMATCH:

YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vcIntegrity thumbprint of  to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vim.eam thumbprint of  to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vlcm.client thumbprint of  to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vsan.health thumbprint of  to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##

In the above, YYYY-MM-DDTHH:MM:SS is the current date/time, and ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:## is the thumbprint being compared to. Note how there is no thumprint being compared, that is, "... thumbprint of   to ...". Note the two spaces between "of" and "to", indicating nothing was printed because the thumbprint was missing.

Compare this to what should be displayed:

YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vcIntegrity thumbprint of YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vim.eam thumbprint of YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vlcm.client thumbprint of YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
YYYY-MM-DDTHH:MM:SS - [operation.check_certificate - check_vcenter_extension_thumbprints] - INFO - Comparing com.vmware.vsan.health thumbprint of YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY to ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##

Note that YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY is the extension thumbprint currently on vCenter (if it exists, which it doesn't in the case of the issue being reported for the KB).

In a properly configured system, for each extension, the reference thumbprint "##:##:..." should be identical to the actual vCenter thumbprint "YY:YY:...". If they are the same, then vCert will report "MATCHES". However, if they are not the same, or if the vCenter thumbprint for the extension is empty, in both cases "MISMATCH" is displayed.

Environment

vCenter 8.0.3 to 9.1 upgrade

Cause

The extension thumbprints are not in fact mismatching but are instead missing. Reporting these as mismatched versus missing is expected behavior in the current version of vCert 6.1.1, so the mismatch report is a false positive. This can be verified by looking at the vCert log for lines like "Comparing com.vmware._extension_name_ thumbprint of to _existing_thumbprint_". Note "thumbprint of to", indicating a missing thumbprint.

Resolution

Use vCert to manually force-push the extension thumbprints/PEM strings into the VCDB, restart services, and rerun vCert to verify the extension thumbprints now match.

 

 

Detection

• From the vCert top menu select "1. Check current certificate status".
• Enter the Single Sign-On administrator account and account password.
• Note in the output:

 

Checking vCenter Extension Thumbprints 
-----------------------------------------------------------------
 com.vmware.vcIntegrity (vpxd-extension) MISMATCH
 com.vmware.vim.eam (vpxd-extension) MISMATCH
 com.vmware.vlcm.client (vpxd-extension) MISMATCH
 com.vmware.vmcam (Authentication Proxy) MATCHES
 com.vmware.vsan.health (Machine SSL) MISMATCH

 

Correction

• From the vCert top menu select "3. Manage certificates".
• From the Manage vCenter Certificates menu select "7. vCenter Extension thumbprints".
• Enter the Single Sign-On administrator account and account password if prompted.
• The extension thumbprints status will be displayed.
• If MISMATCH is displayed for any extensions, you will be prompted to update the extension thumbprints. Enter "y" and press return.
• The Manage vCenter Certificates menu will be displayed again. Press return to return to the main menu.
• From the main menu select "8. Restart services".
• From the Restart VMware Services menu select "1. Restart all VMware services".
• You will be prompted to restart vmware services. Enter y and press return.
• Services will restart (this may take some time).
• Once the services have restarted, from the vCert top menu select "1. Check current certificate status". All thumbprints should now show MATCHES.