vRealize Automation machine provisioning to vSphere cloud accounts fail with error "SSL peer shut down incorrectly"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Deployments and data-collections for vSphere cloud accounts intermittently fail with: "SSL peer shut down incorrectly".
• The certificates for vCenter are valid and have been imported into vRealize Automation by validating the vSphere cloud account, accepting the certificate prompt and saving the cloud account.
• The vRealize Automation provisioning service log located under /services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log contains messages similar to:

2023-07-06T08:00:20.506Z ERROR provisioning [host='provisioning-service-app-6489496d79-hxr94' thread='Thread-59960' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.r.vslm.VslmSession.lambda$getVslmServiceContent$0:100 - https://<vCenterFQDN>/vslm/sdk: Failed to retrieve service content. com.sun.xml.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake 

• On the vCenter instance the envoy.log located under /var/log/vmware/envoy/envoy.log contains messages similar to:

2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] remote https connections exceed > max allowed: 2048
2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] closing connection TCP<IP:Port>, <IP:PORT>

Environment

VMware vRealize Automation 8.x

Cause

vCenter sets a limit on the number of allowed remote https connections.The issue occurs when this limit is exceeded by vRealize Automation.

Resolution

To resolve the issue implement one of the options contained in the Workaround section in vCenter KB article 344920 to increase the number of allowed remote https connections.