vRealize Automation requests fail with the error "Connection reset by peer"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
Various requests fail with the error Connection reset by peer.

Environment

VMware vRealize Automation 8.x

Cause

The maxIdleTime value within Netty connection pools for the provisioning-service is not set.  It is possible that these connections are closed by the underlying networking layer without Netty understanding this change.  When Netty makes subsequent calls to use these connections, the attempt fails with Connection reset by peer.

Resolution

VMware is aware of this issue.  A fix is being considered for a future release.

Workaround:

1. Take snapshot(s) of the vRA node(s) - without including memory.
2. Download the attached set_netty_max_idle.sh script.
3. SSH to one of the nodes in the vRA cluster and upload the downloaded file.
4. Make the script executable

   chmod +x set_netty_max_idle.sh

5. Run the script

   ./set_netty_max_idle.sh

6. Review the output of the script for any errors and validate the final message is [Verify] Verification completed successfully.

   Note:  The script will provide further details to the shell during execution.  If kubectl reports a failure in the command kubectl get nodes for any appliance in the cluster, the script will fail with a verification error.  
   Other errors during execution can include:  Patching failed on one or more nodes, Patch cannot be applied and Patch is not properly applied.

7. Start vRA by running

   /opt/scripts/deploy.sh

8. Remove the script from the node in which it was uploaded

   rm set_netty_max_idle.sh