ESP Workload Automation Subdelay Transmitter Busy

Products

• ESP Workload Automation

Issue/Introduction

Agent Communication Problem Transmitter busy.

SUBDELAY Transmitter busy

The transmitter that passes AFMs (Automated Framework Messages) to the job's Agent is busy with another AFM.

If this status appears for a long time, there may be a problem with the AFM currently being passed by the transmitter. The problem could be due to an Agent not responding or trouble with the connection or transmission. CSF shows the problem in the status field of the associated job.
You can issue the ESPCOM command to display the state of the transmitter.

ESPCOM shows me status of AGENTS and I am slightly confused if the last line is referring to AGENTS or TRANSMITTERS( like XCF).

Environment

Component: ESP WORKLOAD AUTOMATION 
Release:   12.0

Resolution

Identify the problem area or stage:

ESPCOM when used refers to Agent communication only. Has nothing to do with XCF.   

     - If it shows “Agent notified”, it means ESP can take to agent without problem:

      1. Check transmitter.log, if there is any exception error, then the agent has wrong manager communication settings;

      2: Check if the agent spool directory is full.

      3. Check defaultlog_agent.log and runner_os_component.log (with log.level=5 in agentparm), if there is any error. The normal pattern for submitted job in runner_os_component.log should be:

           Received <20101012 14335617+0400 CYBAGENT JavaAgent#tcpip@X550_MANAGER CYBLZ/AGENT21.285/MAIN

           Preparing job CYBLZ/AGENT21.285/MAIN

           Job CYBLZ/AGENT21.285/MAIN starting

          Transmitter: Sending AFM: . . JavaAgent#tcpip@X550_MANAGER OSCOMPONENT ZYBLZ/AGENT21.285/MAIN State EXEC SetStart Status(Executing at CYBAGENT) Jobno(6084)

          Job CYBLZ/AGENT21.285/MAIN has finished

          Transmitter: Sending AFM: . . JavaAgent#tcpip@X550_MANAGER OSCOMPONENT ZYBLZ/AGENT21.285/MAIN State State SUBERROR Cmpc(81011) SetEnd Status

 If the lines with “Transmitter: Sending AFM” don’t show up, then for UNIX agent, it’s possibly  related to IPCS queue.  “ipcs –q”  and “ipcrm -q queue_id”(the related queue_id can be found in runner_os_component.log) can be used to clear up the current queue. And oscomponent.msgqueue may need to be defined and not using the default.

    - If it shows “Connection error” or “transmitter Busy”, it means ESP can’t get proper feedback from agent, it can be caused by:

       1. Agent connection parms (like IP or DNS or port) is defined wrongly in AGENTDEF;

       2. Agent parms, like encryption, ASCII/EBDIC etc, is defined unmatched in AGENTDEF; there will be error messages in ESP JESMSGLG and/or agent receiver.log/defaultlog_agent.log; For R11 AS400 agent, ASCII should be useD.

       3.  There are queued messages with previous wrong settings, issue OPER ESPCOM DEST(agent_name) FLUSH to clear up the queue;

       4. Agent is NOT started, and/or the agent input port is not in Listening status, please check the agent process and its input port;

       5. Check ESP JESMSGLG, if related messages have W at the end (like 1546W), then it’s warning message mostly caused by TIMEOUT; however if they have E at the end (like 1545E), it means definitely an error.

    - If problem occurs AFTER Shadow Manager takes over and MGRADDR command issued:

       1. For AS400 agent, need to manually change the CFGTCP table, remove the old pair of manager name and manager IP address, and add the new pair of manager name with manager IP address.

     2. Agent having natted MANAGER IP Address can’t send messages to ESP until it’s recycled. See more on TEC524064. Use DNS name on MANAGER statement in AGENTDEF table, combined with the IF logic. For example:

     IF SYSNAME='SYSA' THEN DO

         MANAGER NAME(ESP_MANAGER) TCPIP(SYSA.xxxx.COM) 

      ENDDO

      ELSE IF SYSNAME='SYSB' THEN DO

          MANAGER NAME(ESP_MANAGER) TCPIP(SYSB.xxxx.COM)  

      ENDDO

      ELSE IF SYSNAME='SYSC' THEN DO

          MANAGER NAME(ESP_MANAGER) TCPIP(SYSC.xxxx.COM) 

      ENDDO

The MGRADDR command will send the DNS name that is specified on the TCPIP operand of the Manager statement in AgentDef. It will be up to the agent side to resolve it to a numeric address. It implies the availability of a DNS server or/and an appropriate HOST file on the agent side.

- Possible things to check if it’s a TCPIP issue: configuration in AGENTDEF and agentparm.txt are consistent.

       1. If firewall is used in between: the sending ports from both agent and ESP should be open, it can be any available ports; only the receiving port is definite;

       2. If they use DNS name, try with IP address and see if it works; if it is, then the problem is related to DNS server and/or /etc/hosts table on UNIX box. If “TSO ping host_name” from ISPF and “INET Q H host_name” return different IP address, then on ESP STC, //SYSTCPD should be added to point to the proper TCPIP setting.

       3. When DNS name is used for agent address, if warning messages (like 1546W and/or 2214W) show before requests send successfully from ESP, then RESOLVERTIMEOUT in TCPIP.DATA (the one specified on //SYSTCPD in ESP STC) should be decreased from the default 30 seconds to 3. It is used to specify the amount of time the resolver waits for a response while trying to communicate with a name server.  A name of a DNS server is specified by NSINTERADDR statement in TCPIP.DATA. There could be more than one DNS server defined there. If a DNS server  on that list is unreachable, then the resolver waits an amount of time specified by RESOLVERTIMEOUT and then moves to the next one. This can result in visible delays.

       4. If natted or virtual IP address is used on agent and/or ESP side; if yes, try without it and see if it works;

       5. If agent resides on the INACTIVE node of a cluster server; it won’t have fully communication functionality. When cluster is used, AGENT MONITOR should be set only to the agent on ACTIVE node, which can be done easily pointing to the virtual agent only.

- As general approach, please gather following(when nothing above can resolve the problem):

   - Change log.level=8 dynamically and get agent logs with this value; (IP address and port of ESP and agent will show in receiver.log and transmitter.log)

   - Screen shots of the successful pings(and telnet if it’s allowed) from both environments;

   - ESP auditlog and jesmsglg;

   - AGENTDEF table;

   - INET trace(optional);

   - TCPIP trace(like ethereal trace) from agent server(optional).