openssl 3.0.x, bundled with cflinuxfs4, has a bug which causes latency in some apps

Products

• VMware Tanzu Application Service

Issue/Introduction

A customer app built on .NET framework started performing slowly after customer migrated from cflinuxfs3 to cflinuxfs4. We discovered an issue with cflinuxfs4, and we eventually pinpointed the reason was the version of openssl which changed from 1.1 to 3.0.x. The 3.0.x version of openssl has known performance issues in SSL negotiation with multi-threaded apps and this issue has made this unusable for multi-threaded apps that have low latency requirements.
https://github.com/openssl/openssl/issues/17064
There is no indication that this will be fixed in openssl 3.0. openssl 3.3, while not as performant as 1.1, is close and would address the performance issues: https://github.com/openssl/project/issues/100

Environment

Ops Manager 3.0
TAS 4.0.25

 

Cause

This openssl bug is the source of the problem:

https://github.com/openssl/openssl/issues/17064

Resolution

The stack isn't pinning OpenSSL versions, so there is an option to use 3.3, given that it's not installed by default as part of the Jammy release. Tanzu engineers have succeeded in getting a .NET 8 app to load the old openssl 1.1 libs which were pushed with the app.

The trick is to set the CLR_OPENSSL_VERSION_OVERRIDE env var; for example

CLR_OPENSSL_VERSION_OVERRIDE=1.1
CLR_OPENSSL_VERSION_OVERRIDE=3.3