Tunnel "RSA_EAY_PUBLIC_DECRYPT: data too large for modulus" hub error

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

New Tunnel server is not seen in IM with Hub/Robot 9.30 on Windows 2019 on both hubs

Tunnel Server hub.log
hub: SSL handshake start from %TunnelClientIP%/61747 fd=1800: before/accept initialization
hub: SSL state (accept): before/accept initialization
hub: SSL state (accept): SSLv3 read client hello A
hub: SSL state (accept): SSLv3 write server hello A
hub: SSL state (accept): SSLv3 write certificate A
hub: SSL state (accept): SSLv3 write key exchange A
hub: SSL state (accept): SSLv3 write certificate request A
hub: SSL state (accept): SSLv3 flush data
hub: (ssl_server_wait_handshake) - SSL_accept error from host=%TunnelClientIP% fd=1800, err=-1, ssl_err=5(System Call Failure. Check errno.), errno=10054(An existing connection was forcibly closed by the remote host.

Client Hub hub.log
hub: SSL state (connect): before/connect initialization
hub: SSL state (connect): SSLv3 write client hello A
hub: SSL state (connect): SSLv3 read server hello A
hub: SSL error with certificate at depth 0 error: certificate signature failure (7), issuer=...%Tunnel Server IP%
hub: SSL alert (write): fatal: decrypt error
hub: ssl_connect - SSL_connect error (1) on new SSL connection. Socket error (0) - error (The operation completed successfully.
hub: ssl_log_error - SSL_connect error occured, [1] error:0x04067084: rsa routines: RSA_EAY_PUBLIC_DECRYPT: data too large for modulus
hub: TSESS-A-11 could not connect to tunnel %TunnelServerIP%48003 (-2)

Cause

This issue sometimes seen when a third party software interferes with the tunnel hub communication

Environment

Release : 20.1

Component : UIM - HUB

Resolution

Ensure the Hub communication is properly configured on the Firewall and anti-virus exclusion exception for Nimsoft home directory directory is in place.

Additional Information

Firewall Port Reference

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/unified-infrastructure-management/20-1/installing/pre-installation-planning/firewall-port-reference.html