The certificate presented on port 9443 of the HCX Manager does not update after updating the certificate in the HCX admin interface

Products

VMware HCX

Issue/Introduction

Following the certificate update on the HCX Manager appliance, the new certificate is successfully presented on port 443. However, it is not recognized on port 9443.

Environment

VMware HCX

Cause

The automated process fails to apply the updated certificate being used for the HCX appliance-management service.
On HCX manager, login as admin and navigate to the log file /common/logs/admin/web.log, the following error is displayed

2025-02-25 20:14:10.499 UTC [https-jsse-nio-9443-exec-1, , ] ERROR c.v.h.a.c.CertificateManagerRestController- Error in applying >server certificate:
java.lang.Exception: Failed to apply server certificates to HTTP component. Please verify the payload.
at >com.vmware.hybridity.admin.config.CertificateManagerRestController.applyServerCertificates(CertificateManagerRestController.java:29>8)

Resolution

Follow the manual process to update the new certificate being used on port 9443 of the HCX Manager appliance.

Prerequisite:

- We should have the following details of the new certificate that we want to use.

Private key of the certificate
Certificate (CA signed certificate)
Certificate chain (CA signed certificate chain)

NOTE: If the certificate provider does not provide these 3 certificates separately, you will need to pull out each of the above items and create a separate txt file to be able to run the openssl command in step 6.

Steps:

1. SSH to the HCX manager

2. Change to root.

su -

3. Stop the appliance-management service

systemctl stop appliance-management

4. List the /common/appliance-management directory to note the file permissions.

example:

ls -haltr /common/appliance-management/
-r-------- 1 root root 24 Dec 19 05:26 iv_file.txt
drwxr-xr-x 2 root root 4.0K Mar 25 06:40 backup-restore
drwxr-xr-x 35 root root 4.0K Mar 25 06:40 ..
-r-------- 1 admin secureall 2.7K Mar 25 06:40 appl-mgmt-ks.p12.orig
drwxrwxrwx 2 root root 4.0K Mar 25 06:41 ks
-rw-r--r-- 1 postgres postgres 882 Mar 25 06:41 appl-mgmt.cert
-rw-r--r-- 1 postgres postgres 2.7K Mar 25 06:43 appl-mgmt-cert.jks
-r-------- 1 root root 78 Mar 25 06:45 db.properties
-rw-r--r-- 1 root root 113 Mar 25 06:45 vsmvam.db
-r-------- 1 admin secureall 4.2K Mar 25 07:00 appl-mgmt-ks.p12
drwxrwxr-x 4 root appmgmt 4.0K Apr 2 19:24 .

5. Rename /common/appliance-management/appl-mgmt-ks.p12 to something else that is descriptive /common/appliance-management/appl-mgmt-ks.bak<date>

cd /common/appliance-management/
mv appl-mgmt-ks.p12  appl-mgmt-ks.p12.bak4_2_25

6. Use below openssl command to store the certificate details in PKCS12 store

PKCS12 Generation
openssl pkcs12 -export -inkey <privateKey_file.pem> -in <cert_file.pem> -certfile <certChain_file.pem> -out /common/appliance-management/appl-mgmt-ks.p12 -name keyalias -password pass:'<password>' -nomac

NOTE:
When OS is photon without -nomac option you will get an error.

Files used in the above openssl command, will need manually created using the details of the needed individual certificates (Server Certificate, Private Key, and/or CA Certificate chain) Contents of the files should include:

-----BEGIN CERTIFICATE-----

<certificate text>

-----END CERTIFICATE-----

7. Make sure the permissions on the new file matches the original.

example:

ls -haltr /common/appliance-management/
-r-------- 1 root root 24 Dec 19 05:26 iv_file.txt
drwxr-xr-x 2 root root 4.0K Mar 25 06:40 backup-restore
drwxr-xr-x 35 root root 4.0K Mar 25 06:40 ..
-r-------- 1 admin secureall 2.7K Mar 25 06:40 appl-mgmt-ks.p12.orig
drwxrwxrwx 2 root root 4.0K Mar 25 06:41 ks
-rw-r--r-- 1 postgres postgres 882 Mar 25 06:41 appl-mgmt.cert
-rw-r--r-- 1 postgres postgres 2.7K Mar 25 06:43 appl-mgmt-cert.jks
-r-------- 1 root root 78 Mar 25 06:45 db.properties
-rw-r--r-- 1 root root 113 Mar 25 06:45 vsmvam.db
-r-------- 1 admin secureall 4.2K Mar 25 07:00 appl-mgmt-ks.p12
drwxrwxr-x 4 root appmgmt 4.0K Apr 2 19:24 .

- If not, fix the permissions so they match. This example is based on the above sample output from ls -haltr
  chmod 400 appl-mgmt-ks.p12
- Ensure the user and group ownership of the file matches. If not, update the user:group permissions to match.
  chown admin:secureall appl-mgmt-ks.p12

8. Start the appliance-management service

systemctl start appliance-management

9. Validate that the new cert is installed and working.

10. Delete the backed up keystore in this case /common/appliance-management/appl-mgmt-ks.bak<date>, once validation is performed

cd /common/appliance-management/
rm app-mgmt-ks.bak<date>

===============================================================================================

If Rollback is needed

1. Stop the appliance-management service

systemctl stop appliance-management

2. Remove the problematic new cert.

cd /common/appliance-management/
rm appl-mgmt-ks.p12

3. Rename /common/appliance-management/appl-mgmt-ks.p12.bak<date> back to /common/appliance-management/appl-mgmt-ks.p12

cd /common/appliance-management/
mv appl-mgmt-ks.p12.bak<date> appl-mgmt-ks.p12

4. Make sure the permissions on the new file matches the original.

example:
ls -haltr /common/appliance-management/
-r-------- 1 root root 24 Dec 19 05:26 iv_file.txt
drwxr-xr-x 2 root root 4.0K Mar 25 06:40 backup-restore
drwxr-xr-x 35 root root 4.0K Mar 25 06:40 ..
-r-------- 1 admin secureall 2.7K Mar 25 06:40 appl-mgmt-ks.p12.orig
drwxrwxrwx 2 root root 4.0K Mar 25 06:41 ks
-rw-r--r-- 1 postgres postgres 882 Mar 25 06:41 appl-mgmt.cert
-rw-r--r-- 1 postgres postgres 2.7K Mar 25 06:43 appl-mgmt-cert.jks
-r-------- 1 root root 78 Mar 25 06:45 db.properties
-rw-r--r-- 1 root root 113 Mar 25 06:45 vsmvam.db
-r-------- 1 admin secureall 4.2K Mar 25 07:00 appl-mgmt-ks.p12
drwxrwxr-x 4 root appmgmt 4.0K Apr 2 19:24 .

- If not fix the permissions so they match.
  chmod 400 file_name

5. Start the appliance-management service

systemctl start appliance-management