VCSA TLS 1.3 Connection Fails on Port 443 with COMPATIBLE and NIST_2024 TLS Profiles

Products

• VMware vCenter Server

Issue/Introduction

• VCSA server connections over TLS 1.3 fail specifically on port 443 when configured with the COMPATIBLE or NIST_2024 TLS profiles.
• This issue is observed when attempting to establish connections using openssl commands, with TLS 1.3 requests being dropped by the VCSA.
• The openssl s_client output shows the error as "Secure Renegotiation IS NOT supported"

openssl s_client -connect <vCenter-Server-IP>:443 -tls1_3

CONNECTED(00000003)
8772077568:error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol 
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 194 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported ------->>>>>>>> :- It shows Secure Re-negotiation is not supported.
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol : TLSv1.3
    Cipher : 0000

Environment

vCenter Server 8.x

Cause

• TLS 1.3 does not establish a connection over port 443 when either the COMPATIBLE or NIST_2024 TLS profiles are active on the vCenter server.
• As per the documentation vSphere TLS Configuration, this limitation is specific to port 443; TLS 1.3 functions correctly with COMPATIBLE and NIST_2024 profiles on other ports.
• The underlying cause is related to FIPS compliance requirements or conflicts for services (such as the envoy service) running on port 443.

Resolution

• To enable TLS 1.3 functionality over port 443, switch the VCSA TLS profile to COMPATIBLE-NON-FIPS which is FIPS non compliant.
• The switching of TLS profiles can be done by executing the TLS_PROFILE.sh script available from the Broadcom Knowledge Base article Managing TLS Profiles in vCenter 8.0 U3

Additional Information

• After applying the COMPATIBLE-NON-FIPS profile, login to the vCenter server as user root and switch to the Linux bash using the command shell.
• Run the command /usr/lib/vmware-vsr/bin/ssl_scanner --host localhost:443 | less and it should successfully show ciphers for TLS 1.3 enabled.

- version: tlsv1_3
ciphers:
accepted:
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256
- TLS_AES_128_GCM_SHA256
rejected:
- TLS_AES_128_CCM_8_SHA256
- TLS_AES_128_CCM_SHA256
groups:
accepted:
- prime256v1
- secp384r1
- secp521r1
rejected:
- x25519
- x448
- ffdhe2048
- ffdhe3072

 

• It's important to note that while COMPATIBLE-NON-FIPS enables TLS 1.3 on port 443, FIPS-related services (like the envoy service that runs on port 443) will not be FIPS compliant under this profile, despite appearing connected.