ESXi host certificate replacement with Custom certificate fails with error "The private key format is invalid".

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Importing custom SSL certificate for an ESXI host from vSphere Client fails with the error "The private key format is invalid". 
  
  
  
  
• This error is observed after clicking Finish button in the Certificate upload workflow - select Host -> Configure -> Certificate -> Manage with external CA -> Import and Replace -> Upload the certificate -> Next -> Finish.

Environment

• vCenter Server 8.x
• vCenter 9.x
• ESXi 8.x
• ESX 9.x

Cause

• This issue is caused when the Private Key is in PKCS#1 format which starts with the header "-----BEGIN RSA PRIVATE KEY-----".
• For ESXi certificate replacement, the private key header should start with "-----BEGIN PRIVATE KEY-----".

Resolution

Convert the private key to the correct format which starts with the header "-----BEGIN PRIVATE KEY-----" by following below steps:

1. Login to vCenter Server via SSH.
2. Change shell to bash.

Connected to service

       * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

   Command> shell

3. Copy the Private Key to vCenter Server (can use 'vi' editor and copy & paste the private key contents from local desktop).
4. Convert the Private Key format using the command below:
   
   openssl rsa -in private_key.key -out private_key_1.key
   
   
5. Use the converted Private Key (eg. private_key_1.key) for the certificate replacement.