Configuring CA signed certificates for VLSR/or VRMS/or VRS
search cancel

Configuring CA signed certificates for VLSR/or VRMS/or VRS

book

Article ID: 422890

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

This article provides steps to configure Certificate-Based Authentication using CA-signed/Custom certificates.

Symptoms:

  • Unable to use PKCS #12 certificates on versions 9.0.1, 9.0.2, 9.0.3, and 9.0.4.

  • Unable to apply the CA chain certificate.

  • Unable to apply a custom certificate on the appliance.

  • Unable to convert .cer / .crt / .pem certificates into PKCS #12 format for use with the appliance.

  • Unable to successfully create or apply CA-signed certificates with a 4096-bit key size.

  • When attempting to replace the VMware Live Site Recovery appliance certificate with a custom PKCS #12 file, the following error is observed:
    “The certificate is not in the correct format or has a different password.”

  • Unable to generate a CSR using only an FQDN in IP-restricted environments.

Environment

VMware Live Site Recovery 9.0.1.x

VMware Live Site Recovery 9.0.2.x

VMware Live Site Recovery 9.0.3.x

VMware Live Site Recovery 9.0.4.x

Cause

 

  • In many cases, the certificate is successfully applied on the appliance; however, the reconfigure task fails afterward. This typically occurs because the private key is incorrect or inaccessible when applying a custom certificate.

    • When a CSR is generated from the appliance GUI, the private key is automatically generated and stored in the appliance’s default location, and the same key is used during certificate application.

    • When a CSR is generated using the OpenSSL command, the private key is generated externally and is not stored in the appliance’s default location. As a result, the appliance cannot access the private key during certificate application, causing the task to fail.

  • PKCS #12 format is not supported in VLSR versions 9.0.3 and 9.0.4.

  • Applying a PKCS #12 certificate fails when the CSR is generated using OpenSSL, because the associated private key is not present in the appliance’s default location. This limitation applies specifically to appliance-based certificate replacement on versions 9.0.1 and 9.0.2.

  • When attempting to replace the VMware Live Site Recovery appliance certificate with a custom PKCS #12 file on versions 9.0.1 and 9.0.2, using a CSR generated from the UI, the following error may appear:
    “The certificate is not in the correct format or has a different password.”

    • If the password is verified as correct, this error is typically caused by the PKCS #12 file being created with the legacy encryption algorithm RC2-40-CBC, which is not supported.

  • While 4096-bit key size certificates are supported by the appliance, applying such certificates may still fail if the private key is not present in the appliance’s default location, which is required for successful certificate replacement.

 

Resolution

Recommendations :

  • By default, the appliance uses PEM-format certificates. Ensure that all certificates are converted to PEM format before applying them to the appliance.

  • If your environment does not require PKCS #12 certificates, there is no need to use this format. PEM format is fully supported and recommended, unless PKCS #12 is explicitly mandated by your certificate authority or internal security policy.

  • When using a certificate chain with non-PKCS #12 formats:

    • Combine the root CA and intermediate (sub-CA) certificates into a single file.

    • Keep the appliance (leaf) certificate in a separate file.

  • When applying certificates:

    • Only the appliance (leaf) certificate is mandatory.

    • The CA chain is optional during the initial certificate application.

    • Root and intermediate CA certificates can be added after the certificate is applied, if required.

  • If the CSR is generated via the command line (OpenSSL), ensure that the generated private key is moved to the appliance’s default private key location. The appliance must be able to access the private key from this location for the certificate application to succeed.

  • Wherever possible, use the VLSR appliance itself to generate the CSR via the command line and to perform OpenSSL-related operations to avoid compatibility or file placement issues.

Note:

  • For appliance versions 9.0.1 and 9.0.2, the default private key location is: /opt/vmware/dr/conf/

  • For appliance versions 9.0.3 and 9.0.4, the default private key location is: /opt/vmware/etc/va-configurator/

  • The expected private key file name is csrkey.pem.

Steps to Create a CSR Using the OpenSSL Utility on the Appliance

1. Access the Appliance

  • SSH to the appliance.

  • Switch to the root user.

2. Prepare the OpenSSL Configuration

Create the required directory and copy the OpenSSL configuration file:

  • mkdir -p /usr/lib/ssl/
  • cp /etc/ssl/openssl.cnf /usr/lib/ssl/

3. Generate an Initial CSR via the GUI

  • Generate a CSR using the VAMI (GUI) to create the default private key file csrkey.pem in the appliance’s default location.

  • This key will be replaced later with the private key generated using OpenSSL in the following steps.

4. Generate CSR and Private Key Using OpenSSL

Run the following command to generate a new private key and CSR:

  • openssl req -new -nodes -newkey rsa:2048 -keyout vlsr.key -out vlsr.csr -subj "/C=CC/ST=State/L=Locality/O=Organization/OU=OrgUnit/CN=Common_Name_or_FQDN" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf; printf "\n[SAN]\nsubjectAltName=DNS:FQDN,DNS:alias,IP:IP-address")

Replace the following values according to your environment:

  • vlsr.csr → Required CSR file name

  • vlsr.key → Required private key file name

  • C, ST, L, O, OU, CN → Country, State, Locality, Organization, Organizational Unit, and Common Name

  • DNS:FQDN → Hostname / Fully Qualified Domain Name

  • IP:IP-address → Appliance IP address

Notes:

  • The country code (C) must be a two-letter value (for example, US).

  • If an alias is not required, remove DNS:alias.

  • If an IP address is not required, remove IP:IP-address.

  • To generate a 4096-bit key, replace rsa:2048 with rsa:4096.

  • Example:  
    openssl req -new -nodes -newkey rsa:2048 -keyout vlsr.key -out vlsr.csr -subj "/C=US/ST=CA/L=PaloAlto/O=VMware/OU=SRM/CN=vlsr.domain_name.top-level-domain" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf; printf "\n[SAN]\nsubjectAltName=DNS:vlsr.domain_name.top-level-domain,DNS:vlsr,IP:XXX.XXX.X.XX")


5. Convert the Private Key and Replace the Default Key

  • Convert the generated private key to PEM format: openssl rsa -in vlsr.key -out csrkey.pem -outform PEM
  • Copy the key to the appliance’s default private key location:
    • Appliance versions 9.0.1 and 9.0.2 : cp csrkey.pem /opt/vmware/dr/conf/csrkey.pem
    • Appliance versions 9.0.3 and 9.0.4: cp csrkey.pem /opt/vmware/etc/va-configurator/csrkey.pem

6. Obtain the Signed Certificate

  • Use the generated .csr file to request a signed certificate from your Certificate Authority (CA).

7. Handle Certificate Chains (If Applicable)

If a chained certificate is issued:

  • Export the root CA certificate in a separate file.

  • Export any intermediate CA certificates in separate files as well.

8. Apply the Certificate via the GUI

  • Convert a Certificate from CRT, DER, or CER Format to PEM Format

    • Use the following OpenSSL command to convert the certificate to PEM format: openssl x509 -in cert.crt -out cert.pem -outform PEM

    • Note: Replace cert.crt with cert.cer or cert.der, as applicable to your certificate format.

  • Replace the certificate using the VAMI GUI.

  • Apply only the leaf certificate initially (the CA chain is optional at this stage).

  • Refresh the browser and log in again.

  • Verify:

    • The certificate is shown in the browser.

    • The certificate displayed under the Certificate tab in VAMI.
      Both must match.

9. Add CA Certificates (If Required)

  • Add the root and intermediate CA certificates to the appliance if required.

10. Reconfigure the Appliance

  • Reconfigure the appliance to complete the certificate replacement process.

Steps to Convert .cer / .crt Certificates to PKCS #12 Format without legacy algorithms

Important:

  • Conversion to PKCS #12 requires the private key in .key format.

  • This procedure is supported only for versions 9.0.1 and 9.0.2.

Prerequisites

  • A valid certificate file (.cer or .crt)

  • The corresponding private key

  • SSH access to the appliance with root privileges

Private Key Format Considerations

  • If the CSR was generated via the GUI, the private key is created in PEM format (csrkey.pem).

  • If the CSR was generated via OpenSSL, the private key is typically already available in .key format.

  • If the key is in PEM format, it must be converted to .key format before generating the PKCS #12 file.

Step 1: Convert PEM Private Key to .key Format (If Required)

  1. SSH to the appliance and switch to the root user.

  2. Navigate to the default private key location:

    • cd /opt/vmware/dr/conf

    • Convert the PEM key to .key format: : openssl rsa -in csrkey.pem -out /home/admin/csrkey.key

  • The converted private key will be created at: /home/admin/csrkey.key

Step 2: Generate the PKCS #12 Certificate

Run the following command to generate the PKCS #12 file using modern encryption algorithms (non-legacy): openssl pkcs12 -export -inkey csrkey.key -in cert.cer -out cert.p12 -keypbe aes-256-cbc -certpbe aes-256-cbc

Notes:

  • Replace cert.cer with cert.crt if applicable.

  • You will be prompted to set a password for the .p12 file.

  • The cert.p12 file will be generated upon successful completion.

Step 3: Upload the PKCS #12 Certificate

  1. Use SCP to download the generated cert.p12 file to your local system.

  2. Log in to the VAMI interface.

  3. Navigate to: VAMI > Certificate > Change

  4. Select the PKCS #12 option.

  5. Upload the cert.p12 file and provide the password. Click on change.

Step 4: Verify Certificate Application

  • After Chaning the certificate, you may see an error message.
    This behavior is expected when applying PKCS #12 certificates.

  • Refresh the browser page.

  • Log in to the VAMI interface again.

  • Verify the certificate:

    • From the browser certificate details

    • From VAMI > Certificate

  • Ensure that the Issuer and Fingerprint match the uploaded certificate. If the details match, the certificate has been successfully replaced.

  • Note: If the PKCS #12 certificate was already created using a legacy encryption algorithm, extract the leaf certificate from the file and retrieve the corresponding private key from the appliance. Then, recreate the PKCS #12 file using the steps outlined above to ensure it is generated with a supported, non-legacy encryption algorithm.




Additional Information

The most common SSL certificates are defined by X.509. The digital certificates have different formats. Here’s a brief overview of several common formats of SSL certificates:

PEM: private key or certificate

CER: only certificate

DER: only certificate

CRT: only certificate

PFX: private key and certificate

P12/P7B: private key and certificate

JKS: private key and certificate

KEY: public key or private key

PEM Format

The PEM format is the most common format that Certificate Authorities issue certificates in. PEM certificates usually have extensions such as .pem, .crt, .cer, and .key. They are Base64 encoded ASCII files and contain "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" statements. Server certificates, intermediate certificates, and private keys can all be put into the PEM format.

Apache and other similar servers use PEM format certificates. Several PEM certificates, and even the private key, can be included in one file, one below the other, but most platforms, such as Apache, expect the certificates and private key to be in separate files.

DER Format

The DER format is simply a binary form of a certificate instead of the ASCII PEM format. It sometimes has a file extension of .der but it often has a file extension of .cer so the only way to tell the difference between a DER .cer file and a PEM .cer file is to open it in a text editor and look for the BEGIN/END statements. All types of certificates and private keys can be encoded in DER format. DER is typically used with Java platforms. The SSL Converter can only convert certificates to DER format. If you need to convert a private key to DER, please use the OpenSSL commands on this page.

PKCS#7/P7B Format

The PKCS#7 or P7B format is usually stored in Base64 ASCII format and has a file extension of .p7b or .p7c. P7B certificates contain "-----BEGIN PKCS7-----" and "-----END PKCS7-----" statements. A P7B file only contains certificates and chain certificates, not the private key. Several platforms support P7B files including Microsoft Windows and Java Tomcat.

PKCS#12/PFX Format

The PKCS#12 or PFX format is a binary format for storing the server certificate, any intermediate certificates, and the private key in one encryptable file. PFX files usually have extensions such as .pfx and .p12. PFX files are typically used on Windows machines to import and export certificates and private keys.

When converting a PFX file to PEM format, OpenSSL will put all the certificates and the private key into a single file. You will need to open the file in a text editor and copy each certificate and private key (including the BEGIN/END statements) to its own individual text file and save them as certificate.cer, CACert.cer, and privateKey.key respectively.

  1. Convert certificate and private key in PEM format to PFX format : openssl pkcs12 -export -out certificate.pfx -inkey privkey.pem -in cert.pem
  2. Convert certificate in CRT, DER or CER format to PEM format.: openssl x509 -in cert.crt -out cert.pem -outform PEM
  3. Convert private key in KEY format to PEM format.: openssl rsa -in private.key -out private.pem -outform PEM
  4. Convert PEM to DER : openssl x509 -outform der -in certificate.pem -out certificate.der
  5. Convert PEM to P7B:  openssl crl2pkcs7 -nocrl -certfile certificate.cer -out certificate.p7b -certfile CACert.cer
  6. Convert PFX to PEM : openssl pkcs12 -in certificate.pfx -out certificate.cer -nodes

 

OpenSSL - Private and Public Key check

A very common need before deploying the certificate to the server is to compare the mutual match of the private key and the issued certificate. These are always a few keys that must belong together. Otherwise, the certificate would not be installed, or worse, a key error could cause the site to malfunction. The interconnection also applies between the private key and the certificate request (CSR).

Keys verification

A sequence of two OpenSSL commands separated by a "|" character is used to compare keys resulting in a character string that must be the same for both keys. In this case, the tested keys belong together.
The following commands should be entered one after the other to generate separate outputs. The output is an md5 hash (fingerprint) of the key, it can be replaced by the sha256 algorithm, which, however, gives a longer string of characters. You can name the file extensions as you need and you are used to, for example .csr, .crt, .txt.

Comparison of the private key and the issued certificate:

  • openssl rsa -modulus -in private.key -noout | openssl md5
  • openssl x509 -modulus -in cert.txt -noout | openssl md5

Comparison of private key and certificate request:

  • openssl rsa -modulus -in private.key -noout | openssl md5
  • openssl req -modulus -in domena.csr -noout | openssl md5

Each command results in a character string. If the output of each command matches, the keys belong together. If the outputs do not match, the keys do not match and the certificate cannot be installed. The only solution is to generate new keys.

 

Display information of the issued certificate

Use the x509 command to check the issued certificate and its information. This can verify that the information in the certificate is correct and matches your private key.

  • openssl x509 -text -in cert.cer -noout

Replacing VMware Live Site Recovery appliance's custom certificate with PKCS#12 fails

Powered by Wolken Software