Dollar Universe Manager for SAP cannot open connections to SAP system during some time.
As a result, some Jobs that are completed in SAP continue to be shown as Running in Dollar Universe side.
Example of errors in universe.log:
|INFO |X|SAP|pid=p.t1| uxsap_connect | RFC connection fails for SAP system [COMPANY/*/MANAGEMEN_UNIT*]
|ERROR|X|SAP|pid=p.t2| uxsap_connect | uxsap_open_comm error: unable to open the RFC handle
|INFO |X|SAP|pid=p.t3| uxsap_connect_with_retry | Trying a new connection in 10 seconds... Attempt 2/2
After the first occurrence, the UXSAP_NB_CONNECT_RETRY variable was set to 5, but the issue re-occurred again.
Network team analysis provided no network issues that would seem the cause of the connectivity issue.
In the dev_rfc.trc file that is written in the log/rfc or bin_sap folder, the following kind of errors appeared during this period of time:
**** Trace file opened at 2021-05-12, 22:18:15 Romance Daylight Time
RFC Library Release: 721 Patch Level 50, Compiled on Aug 21 2018 00:02:00
CPIC library: 721.2018.08.17 version 3, NI library: 40, Kernel Release: 721 Patch Level 0
Current working directory: C:\Windows\System32, Program: UniSapManager
Hardware: PC with Windows NT 8x AMD64 Level 6 (Mod 85 Step 4), Operating_system: Windows NT 6.3
Error during connect:
LOCATION SAP-Gateway on host server.domain.com / sapgw00
ERROR timeout during accept
TIME Wed May 12 22:18:15 2021
RELEASE 753
COMPONENT SAP-Gateway
VERSION 2
RC 751
MODULE D:/depot/bas/753_REL/src/krn/si/gw/gwxxrd.c
LINE 5804
COUNTER 83884
<...>
Error during connect:
LOCATION SAP-Gateway on host server.domain.com / sapgw00
ERROR timeout during accept
TIME Thu May 13 06:01:43 2021
RELEASE 753
COMPONENT SAP-Gateway
VERSION 2
RC 751
MODULE D:/depot/bas/753_REL/src/krn/si/gw/gwxxrd.c
LINE 5804
COUNTER 2
Release : 6.x
Component : DOLLAR UNIVERSE
Sub-Component: Manager for SAP
Specific Environment:
Manager for SAP with multiple SAP instances declared (uxjcs.ini to be sent)
SAP System issue
First of all, dev_rfc.trc should be looked at for troubleshooting SAP related issues.
This log can be found inside the subfolder rfc of the log folder.
Look for potential error messages around the time of the connectivity issues, as the SAP RFC library will log here the reason of the errors.
In this case, the error was:
ERROR timeout during accept
When the error is identified, SAP Administrators should be contacted and asked to perform an investigation about the related SAP System availability during the time of the issue.
The SAP System log (transaction SM21) should be their starting point, looking for potential errors and a check of the usage of the disk/memory/ dialog work processes.
On this case, most probable cause was that SAP Dialog Work Processors werre very busy during those times and hence our binaries cannot connect to it.
In case your SAP administrators cannot find the root cause, please open a case with SAP Support to perform a more deep root cause analysis.
On this case, a SAP analysis confirmed that the issue was caused by a SAP System issue.
Advice to avoid reoccurrences:
As it all points to a saturation at SAP side, reduce the load during these peak times and/or increase the period until a new connection is retried via the variable UXSAP_NB_CONNECT_RETRY or set the variable AGT_WAIT_CN_KEEP to No close Dollar Universe SAP connection when Job is in status Running.