Error "EBADF-Bad file descriptor" in log

Products

• VMware Smart Assurance

Issue/Introduction

Domain or broker crash observed, error in respective log has:

[May 12, 2025 at 1:39:01 PM GMT+1 +360ms] t@1174337280 HTTP #17
CI-E-EWHILE-While executing function "fillRead"
 CI-EFLOWID-For flow CI_FlowBufferedHead_U [Flow in negotiations Acceptor Buffered Head] HEAD|BUFFERED @0x00007f530482e5e0
  . Read buffer, 0 bytes available of 2145
 . ?3?0000000000000000 0000000000000000 <^|0000000000000000 0000000000000000
  . Write buffer, 0 bytes written of 2048
  . ?3?[^0000000000000000 0000000000000000 0000000000000000 0000000000000000
  . ->CI_FlowTCP_U [Flow in negotiations Accepted physical flow] IN_FLOW|PHYSICAL|LOCAL @0x00007f52ec07cf20
  . *:v4:426 KS N/A, KR N/A
  . Open fd=1035, conn May 12, 2025 at 1:39:01 PM GMT+1, disc N/A,
  . 127.0.0.1:426 -> 127.0.0.1:39528, tmo 0 00:00:15 N/S 1/0
 CI-EWHILEREAD-After reading "0" bytes of "1" (CI_FlowTCP_U)
 CI-EWHILE-While executing function "doSelect"
 <SYS>-EBADF-Bad file descriptor; in file "/work/redcurrent/DMT-10.1.12.0/16/smarts/clsapi/ci_flow.c" at line 1947

Environment

Smarts - 10.1.x

Cause

The error "Bad file descriptor" usually means that program is trying to read from, write to, or close a file or socket that:

• Was never opened,

• Was already closed,

• Or is otherwise invalid (corrupted, wrong permissions, etc.).

Several open connections to domain/service or no graceful shutdown of opened connections add load to domain/service, eventually leading to crash. 

Resolution

Identify which process(es) is/are source of crash. This can be done by various ways:

• Check connections against the service port: 

  netstat -tnalp | grep <port of service crash>

  This will list all connections like ESTABLISHED, LISTEN, CLOSE_WAIT and TIME_WAIT.
• Collect /var/log/messages, look for SEGFAULT, SIGSEGV of crashed service. 
• Collect client dump as 

  dmctl -s <domain> exec dmdebug --clients

  If needed, navigate output to file. Look for fd= which indicates open connection, this should not exceed default fd= provided to a service during startup. This is set by OS to a service, by default it would be 1024.
• Collect Core dump of crashed service.
  Ensure that core is set to unlimited as 
  

  ulimit -c unlimited