Gen DTU very slow in hybrid cloud environment
Article ID: 227376
Updated On:
Products
Issue/Introduction
Experiencing extremely slow performance (5 minutes/step) with the Gen 8.6 Diagram Trace Utility (DTU) when running in a hybrid cloud environment i.e.
Gen application on AWS Linux server connecting to DTU on local in-house PC.
The networking performance between AWS and the local internal network has been tested and is very good.
Have applied several config changes which have had minimal performance improvements:
EFS (Elastic File System) throughput increased
EFS storage upgraded to higher tier from default tier
Changed from EFS mount to local disk (Copied /appl and sub directories from EFS mount to local disk)
Disabled antivirus on /appl and /home directories
Have other customers that have moved to a cloud environment encountered any DTU performance issue?
Environment
Release : 8.6
Component : CA Gen Diagram Trace
Cause
The DTU console log repeatedly showed "Failed to establish Control Connection" messages after each session was initiated:
+++
Listening started, port: 1250
Initiating session with "/nnn.nnn.nnn.nnn:27893"
Failed to establish Control Connection with "127.0.0.1:57860"
*** Not all debugging features will be available.
...
+++
The sequence of events for the DTU connection is:
- The remote Gen application will attempt to connect to the DTU's listening port and at the same time, it will open a listening port using the first available port (ephemeral port) on its server. It will send this information to the DTU.
- The DTU will accept the connection from the Gen application which can be referred to as the "Trace Channel".
- The DTU will use the ephemeral port information sent by the Gen Server and attempt to connect to it, which if successful is referred to as the "Control Channel". The "Control Channel" is meant to speed up the tracing process and is called the "Smart" mode. If this channel is not created the trace will still take place, but possibly in a slower manner.
Resolution
The available/ephemeral port used on the server cannot be specified as a fixed value. Therefore the server firewall needs to be open for all possible ephemeral ports to allow the Control Connection to succeed. For further details see KB article Gen DTU "Failed to establish Control Connection" using remote application
An additional resolution for this particular occurrence was also needed:
In the hybrid cloud, the developers had configured the Linux server environment variable:
C_TRACE_HOST=localhost
That meant the Control Connection was trying to be made to the local PC (127.0.0.1) (message "Failed to establish Control Connection with "127.0.0.1:57860"") instead of the Linux server running the Gen application.
The problem was resolved after changing the Linux server environment variable:
C_TRACE_HOST=<Linux server hostname>
Additional Information
Feedback
