Couldn't establish a connection to the VM web console

Products

• VMware vCenter Server

Issue/Introduction

After updating to VC 9.x VM web console does not work through vcenter, but console will work via host ui gui.

Environment

vc 9.0.x

Cause

ESX host certificate has been improperly formatted and it using 76 characters per line versus the expected 64 characters per line

Per https://datatracker.ietf.org/doc/html/rfc7468

Generators MUST wrap the base64-encoded lines so that each line
consists of exactly 64 characters except for the final line, which
will encode the remainder of the data (within the 64-character line
boundary), and they MUST NOT emit extraneous whitespace.  Parsers MAY
handle other line sizes.  These requirements are consistent with PEM
[RFC1421].

When attempting to view the host certificate via web browser it will look correct.

When a web console connection is made unsuccessfully you may see the log message below.

/var/log/vmware/envoy/envoy.log
YYYY-MM-DDTHH:MM:SS.458+00:00 error envoy[2486] [Originator@6876 sub=connection] Failed to load trusted CA certificates from <inline>
YYYY-MM-DDTHH:MM:SS.463+00:00 info envoy[2808] [Originator@6876 sub=connection] [Tags: "ConnectionId":"531014"] remote address:###.###.###.###:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end-----BEGIN CERTIFICATE-----

 

Resolution

To confirm if certificate is properly format the below can be run on an ESX host.

awk '{ print length }' /etc/vmware/ssl/rui.crt | sort -u

Output:
25
27
52
64 <- 64 chars per line

 

This is expected result, if 76 then the certificate is in the wrong format.

 

-The following command will print out the hash with 64 characters per line. 

openssl x509 -in /etc/vmware/ssl/rui.crt

 

-Backup and replace the rui.crt with fixed format

-Restart services on ESX host to apply changed certificate.

Additional Information

Per https://datatracker.ietf.org/doc/html/rfc4648#section-4

3.1.  Line Feeds in Encoded Data

   MIME [4] is often used as a reference for base 64 encoding.  However,
   MIME does not define "base 64" per se, but rather a "base 64 Content-
   Transfer-Encoding" for use within MIME.  As such, MIME enforces a
   limit on line length of base 64-encoded data to 76 characters.  MIME
   inherits the encoding from Privacy Enhanced Mail (PEM) [3], stating
   that it is "virtually identical"; however, PEM uses a line length of
   64 characters.  The MIME and PEM limits are both due to limits within
   SMTP.

   Implementations MUST NOT add line feeds to base-encoded data unless
   the specification referring to this document explicitly directs base
   encoders to add line feeds after a specific number of characters.