Gen Java EJB server not tracing to DTU

Products

• Gen
• Gen - Run Time Distributed

Issue/Introduction

Running a Gen 8.6 Java EJB server under WebSphere Application Server. After calling the EJB Web Service from a Gen C client the server is not tracing to the remote Diagram Trace Utility (DTU).

The code has been generated with trace and the ear file Assemble step for the Trace tab had the "Enable Diagram Trace" checkbox set with correct DTU hostname/port per EAR File Assembling > Tracing
The installed application "Environment entries for EJB modules" entry also reflects that:

Environment

Gen 8.6 EJB server

Resolution

It is suspected there is a network firewall preventing the opening of the DTU connection.

If run telnet on the Application Server machine it is possible to verify if a connection is possible with the running DTU using the command:
telnet DTU_ip_address 4567
If there is a firewall problem the command will hang and eventually time out e.g. "Connection timed out" or "Connect failed" depending on the Application Server OS.
The telnet command may or may not be installed depending on the Application Server OS platform e.g.
If it is running under Linux "yum install telnet" will install the telnet package
If it is running under Windows the "Telnet Client" can be installed from Control Panel/Programs and Features > "Turn Windows features on or off" which for a Windows Server will redirect to Server Manager > "Add Roles and Features Wizard" ("Telnet Client" is listed under Features)

For this reported scenario, the WebSphere Application Server was running under Linux and to enable the connection to the DTU port on the user's local laptop a putty ssh session was configured to the Linux server and a tunnel was added for the Remote Port Forwarding i.e. Source port 4567 (Remote) and Destination DTU_hostname:4567. 



The value of the "Environment entries for EJB modules" entry was also set to localhost:4567, so the connection to the DTU used localhost:4567 and was forwarded to  DTU_hostname:4567. The tracing was then successful.
NOTE: The putty ssh session needs to remain active for this to work.

Additional Information

TecAdmin.net,: How to Set Up SSH Tunnel with PuTTY
I/O FLOOD: PuTTy SSH Tunneling: A Step-by-Step Guide

In addition to the telnet check for the DTU port, there is an additional connection involved which is described in this article: Gen DTU "Failed to establish Control Connection" using remote application
That article was written for a Gen C server on Solaris but the same principles apply to a Gen Java server i.e.
===
The sequence of events for the DTU connection is:
- The remote Gen application will attempt to connect to the DTU's listening port and at the same time, it will open a listening port using the first available port (ephemeral port) on its server. It will send this information to the DTU.
 - The DTU will accept the connection from the Gen application which can be referred to as the "Trace Channel".
 - The DTU will use the ephemeral port information sent by the Gen Server and attempt to connect to it, which if successful is referred to as the "Control Channel". The "Control Channel" is meant to speed up the tracing process and is called the "Smart" mode. If this channel is not created the trace will still take place, but possibly in a slower manner.

The available/ephemeral port used on the server cannot be specified as a fixed value. Therefore the server firewall needs to be open for all possible ephemeral ports to allow the Control Connection to succeed.
===
Per above if the "Control Channel" cannot be created the tracing should still work, but slower.

The second channel was confirmed by some netstat output Support captured from their tests.
The WebSphere Application Server Gen EJB application "Environment entries for EJB modules" entry was set to use local_ip for the DTU connection.

1. Running a Gen C client from the same machine where DTU is running (local_ip) and calling the EJB Web service running on WebSphere port 9080 (appserver_ip):
After the server PStep is called the DTU shows the server PStep in trace and the connections from appserver_ip are:
C:\>netstat -na|find "appserver_ip"
  TCP    local_ip:4567          appserver_ip:60404    ESTABLISHED
  TCP    local_ip:50461        appserver_ip:9080     ESTABLISHED
  TCP    local_ip:50462        appserver_ip:60405    ESTABLISHED
  
2. Running SOAPUI from any other Windows machine and calling the EJB Web service:
After the server PStep is called the DTU shows the server PStep in trace and the connections from appserver_ip are:
C:\>netstat -na|find "appserver_ip"
  TCP    local_ip:4567          appserver_ip:60431    ESTABLISHED
  TCP    local_ip:50495        appserver_ip:60432    ESTABLISHED
In this instance, checking what process had ports 60431 and 60432 open on the appserver_ip and they were both opened by the WebSphere java.exe process.

So in summary, for optimum tracing the firewall needs to be open between the Application Server machine and the DTU machine for any possible ephemeral port that could be used and not just for DTU side port 4567.