vCenter RDU/NDU Upgrade fails with "Failed to initialize replication of the vCenter Server Appliance for the upgrade process" in an environment where the root certificate was automatically renewed

Products

• VMware vCenter Server

Issue/Introduction

• vCenter RDU/NDU Upgrade fails with the following error:

  Failed to initialize replication of the vCenter Server Appliance for the upgrade process

  
  

• The following log entry is recorded in /var/log/vmware/vlcm/vlcm.log:

  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 81482 [vc@4413] [controller/controller.go:250] Controller.Process(op:execute peer:-, pool:-, label:PHASE:DbReplication) failed: com.vmware.vapi.std.errors.error
  

• Extracting the target log bundle (.tgz) saved under /var/log/vmware/vlcm/ and checking storage/log/vmware/vlcm/vlcm.log reveals the following errors:

  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [exec/exec.go:126] [ObjectID=45] Run command [/opt/vmware/vpostgres/16/bin/pg_basebackup --dbname host=<IP_ADDRESS> port=5432 user=replicator sslmode=verify-ca sslrootcert=/storage/db/vpostgres_ssl/root_ca.pem connect_timeout=10 --checkpoint fast --pgdata /storage/db/vpostgres --wal-method stream --waldir /storage/dblog/vpostgres/pg_xlog --password --progress --no-manifest] failed (duration: 32ms)
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:19] [ObjectID=45] pg_basebackup failed
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:45] [ObjectID=45] ERR:001 exit status 1
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:53] [ObjectID=45] ERR:002-
  YYYY-MM-DDTHH:mm:ss.SSSZ INFO vlcm 23150 [vc@4413] [retry/retry.go:90] [ObjectID=45] retry: cancelled after 1 retries
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:45] [ObjectID=45] OUT:001 Password:
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:45] [ObjectID=45] OUT:002 pg_basebackup: error: connection to server at "<IP_ADDRESS>", port 5432 failed: SSL error: certificate verify failed
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [logger/multiline.go:53] [ObjectID=45] OUT:003-
  YYYY-MM-DDTHH:mm:ss.SSSZ ERROR vlcm 23150 [vc@4413] [engine/engine.go:1029] [ObjectID=45] wfengine: execute [<45:A> dbRepl.Backup] error: com.vmware.vapi.std.errors.error
  YYYY-MM-DDTHH:mm:ss.SSSZ WARNING vlcm 23150 [vc@4413] [dbreplication/pgbasebackup.go:416] [ObjectID=45] pgBaseBackupOutput: failed to parse: [pg_basebackup: error: connection to server at "<IP_ADDRESS>", port 5432 failed: SSL error: certificate verify failed]
  

• On the source vCenter, the current VMCA root certificate (/var/lib/vmware/vmca/root.cer) is not present in the vPostgres certificate cache (/storage/db/vpostgres_ssl/root_ca.pem).

  • Checking the VMCA root certificate:

    # openssl x509 -fingerprint -sha1 -noout -in /var/lib/vmware/vmca/root.cer
    sha1 Fingerprint=03:D9:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:4C:AE
    

  • Checking the vPostgres certificate cache:

    # awk '/-----BEGIN CERTIFICATE-----/{cert=""} {cert=cert $0 "\n"} /-----END CERTIFICATE-----/{cmd="openssl x509 -noout -fingerprint -sha1"; printf "%s", cert | cmd; close(cmd)}' /storage/db/vpostgres_ssl/root_ca.pem
    sha1 Fingerprint=F8:09:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:84:EF
    sha1 Fingerprint=A7:F1:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:6C:94
    

Environment

vCenter Server 9.x

Cause

This issue stems from a stale certificate cache in the vPostgres service originating from a previous upgrade.

During a prior vCenter RDU/NDU upgrade, the system automatically renews the root certificate if its remaining validity is less than two years.
However, because the vPostgres service starts before this renewal process, it does not cache the newer certificate at that time.
If the vPostgres service is not restarted after that previous upgrade, the cache file (/storage/db/vpostgres_ssl/root_ca.pem) remains unupdated and lacks the newly renewed root certificate.
As a result, when a subsequent RDU/NDU upgrade is attempted, the target vCenter fails to establish an SSL connection to the source vCenter due to a certificate verification error.

Resolution

Restarting the services on the source vCenter Server resolves the issue.

1. Connect to the source vCenter Server via SSH.
2. Execute the following commands to restart all services:

   service-control --stop --all
   service-control --start --all
   

3. Retry the upgrade process.