vpxd service failed to start as it couldn't validate certificate.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

vpxd service fails to start as it couldn't validate the certificate.

From the vcenter server logs we can see the below entry :

vpxd.log :-

YYYY-MM-DDTHH:MM:SS info vpxd[7EFD8E4B5800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
YYYY-MM-DDTHH:MM:SS error vpxd[7EFD8E4B5800] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: 3C:3D:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:6A:98
--> ExpectedThumbprint:
--> ExpectedPeerName: localhost
--> The remote host certificate has these problems:
-->
--> * Host name does not match the subject name(s) in certificate.)
--> [context]zKq7AVECAAAAALtNpwANdnB4ZAAATHorbGlidm1hY29yZS5zbwAAHiQbAD5yGABe8RsA7XAiAPg9IgAvQiIAn/kjAAvFIwDyxyMAA9MrAdRzAGxpYnB0aHJlYWQuc28uMAACvY4ObGliYy5zby42AA==[/context]
YYYY-MM-DDTHH:MM:SS error vpxd[7EFD8E4B5800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

Environment

VMware vCenter Server Appliance 6.x
VMware vCenter Server Appliance 7.x

Cause

• vCenter server's certificate were issued by an intermediate CA which in turn was signed by an Internal Root CA.
• Intermediate and Root CA's root certificate was published twice with different aliases in the TRUSTED_ROOTS store.

Resolution

Note: Before carrying out the following procedure, ensure you have a valid back up of all the nodes.

• Identify the aliases of the certificate which has been added multiple times within the TRUSTED_ROOTS store: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep -i serial -A 1
  
          Serial Number:
              d1:c1:##:##:##:##:1d:47
  --
          Serial Number:
              c5:d7:##:##:##:##:68:a6
  --
          Serial Number:
              59:33:##:##:##:##:##:##:##:##:##:##:##:##:a3:ea
  --
          Serial Number:
              3f:00:##:##:##:##:##:##:##:##:##:##:##:##:00:03
  --
          Serial Number:
              59:33:##:##:##:##:##:##:##:##:##:##:##:##:a3:ea
  --
          Serial Number:
              3f:00:##:##:##:##:##:##:##:##:##:##:##:##:00:03
  --
          Serial Number:
              d4:ce:##:##:##:##:cb:77
  --
          Serial Number:
              e1:ff:##:##:##:##:6c:32
• In this scenario, we can see double entries for two serial numbers
  
  Serial Number:  59:33:##:##:##:##:##:##:##:##:##:##:##:##:a3:ea
  Serial Number:  3f:00:##:##:##:##:##:##:##:##:##:##:##:##:00:03
• Check for different aliases for the same certificate and make a note of it 
  
  Example:-
  Alias : xxxxxxx.crt   <---------------------- Different Alias
  Entry type :    Trusted Cert
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              59:33:##:##:##:##:##:##:##:##:##:##:##:##:a3:ea
      Signature Algorithm: sha256WithRSAEncryption
  
  Alias : xxxxxxxxxxxxxxxxx <--------------- Different Alias
  Entry type :    Trusted Cert
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              59:33:##:##:##:##:##:##:##:##:##:##:##:##:a3:ea
      Signature Algorithm: sha256WithRSAEncryption
• Export the certificate to a file: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias_of_the_cert(which you want to unpublish)> --output /tmp/extra_root.crt
• Unpublish the certificate which you have exported to file: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/extra_root.crt
• Run the first command to confirm that we don't have multiple entries for the same certificate: /usr/lib/vmware-vmafd/bin/vecs-cli entry  list --store TRUSTED_ROOTS --text | grep -i serial -A 1
• Start the vpxd service: service-control --start vmware-vpxd