ESXi/ ESX host cannot be reconnected to VirtualCenter after installing SSL certificates on the Host

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• After changing the SSL certificate on ESX 3.5 hosts and VirtualCenter 2.5, you cannot connect to the hosts using VirtualCenter
• You see the error:
  
  failed to install virtual center agent services, login failed due to bad user name and password
  
• You can access the hosts directly using VMware Infrastructure (VI) Client
• /var/log/vmware/vpx/vpxa.log has errors similar to:
  
  [2010-03-23 15:31:38.272 0xf7bda6c0 error 'App'] [VpxdCertificate] Failed: unrecognized file format: /etc/vmware/ssl/rui.crt
  

Environment

VMware vCenter Server 4.0.x
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Installable
VMware VirtualCenter 2.5.x
VMware ESXi 4.0.x Embedded
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.0
VMware ESXi 3.5.x Installable
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Installable
VMware ESX 4.1.x
VMware vCenter Server 4.1.x
VMware vCenter Server 5.0.x

Resolution

This issue may occur if the header of the rui.crt file has hidden control characters or non-printable characters that VirtualCenter does not recognize.

To determine if this issue is caused by the rui.crt file:

1. Login as root to the ESX host using an SSH client.
2. In the /var/log/vmware/vpx/vpxa.log and look for errors similar to:
   
   unrecognized file format: /etc/vmware/ssl/rui.crt
   
   These errors indicate that VirtualCenter does not recognize the rui.crt file.

To resolve this issue, you must create a new rui.crt file.

To create a new rui.crt file:

1. Open the /etc/vmware/ssl/rui.crt using a text editor and copy its contents from:
   
   -----BEGIN CERTIFICATE-----
   
   TO
   
   -----END CERTIFICATE-----
   
   Note: Do not copy any text that comes before -----BEGIN CERTIFICATE-----.
   
   
2. Create a new file named rui1.crt and paste content you copied in step 1 into the file.
3. Run the following commands:
   
   # mv /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui.crt.old
   # mv /etc/vmware/ssl/rui1.crt /etc/vmware/ssl/rui.crt
   # service vmware-vpxa start
   
4. Start the VirtualCenter Agent Service. For more info see Restarting the Management agents on an ESX Server (1003490).
5. Add the ESX host to Inventory.

• This issue also occurs when the rui.crt file has windows line terminators (CRLF) in it. This can be checked by running the command:
  
  cat -v /etc/vmware/ssl/rui.crt
  
• Windows line terminators (CRLF) which will show up as ^M characters can be removed running the command:
  
  tr -d '\r' < /etc/vmware/ssl/rui.crt > /etc/vmware/ssl/rui-fixed.crt && mv /etc/vmware/ssl/rui-fixed.crt /etc/vmware/ssl/rui.crt

Additional Information

To view the characters that prevented VirtualCenter from recognizing the old rui.crt file, run the diff command and compare the new file with the old file:

# diff rui.crt rui.crt.old

The output appears similar to:

1,32c1,32
-----BEGIN CERTIFICATE-----
MIIFmTCCBIGgAwIBAgIKH3SyyAAAAAAHOzANBgkqhkiG9w0BAQUFADBYMRIwEAYK
CZImiZPyLGQBGRYCY2ExFTATBgoJkiaJk/IsZAEZFgVjY2hzYTETMBEGCgmSJomT
8ixkARkWA2ludDEWMBQGA1UEAxMNQ0NIU0EgUm9vdCBDQTAeFw0wODA0MjEyMDE2
MzVaFw0xMDA0MjEyMDE2MzVaMIGEMQswCQYDVQQGEwJDQTEQMA4GA1UECBMHT250
-----END CERTIFICATE--------
-----BEGIN CERTIFICATE-----
MIIFmTCCBIGgAwIBAgIKH3SyyAAAAAAHOzANBgkqhkiG9w0BAQUFADBYMRIwEAYK
CZImiZPyLGQBGRYCY2ExFTATBgoJkiaJk/IsZAEZFgVjY2hzYTETMBEGCgmSJomT
8ixkARkWA2ludDEWMBQGA1UEAxMNQ0NIU0EgUm9vdCBDQTAeFw0wODA0MjEyMDE2
MzVaFw0xMDA0MjEyMDE2MzVaMIGEMQswCQYDVQQGEwJDQTEQMA4GA1UECBMHT250
-----END CERTIFICATE-----