Unable to Connect to CLI Shell Server with "Connection Timed Out" Error

Products

• VMware Avi Load Balancer

Issue/Introduction

• When attempting to access the shell for the Avi Controller, users receive the following error message:
  

  admin@Controller-IP:~$ shell
  Login: admin
  Password:
  
  Please verify that the CLI shell server is enabled
  [Errno 110] Connection timed out
  You can launch /opt/avi/python/bin/cli/bin/shell.py to enable remote CLI shell

• This error prevents users from accessing the controller's command-line interface (CLI) through the remote shell connection.

Environment

• All version of VMware Avi Load Balancer
• Component: Avi Shell Server

Cause

The connection failure occurs due to timing and synchronization issues between the CLI shell client and server. There are two primary scenarios that cause this error:

Scenario 1: Service Restart Timing Issue

• When the clishell_server.service is restarted, the Python socket server requires approximately 10 seconds to fully initialize and become operational.
• If a client attempts to connect during this initialization window, the connection fails because the server is not yet ready to accept connections.
• The error message appears even though the service is starting up normally.

 

Scenario 2: Interrupted Data Transmission (Primary Cause)

• The CLI client collects and sends client information in JSON format to the server during connection establishment.
• The JSON payload typically starts with data such as: {"session_id": "...", "conn_type": "stdio", ...}
  When the connection is interrupted during transmission (due to network issues, service restart, or timing problems), only partial data may be transmitted.
• In some cases, only the first byte (such as 'G' from a field like "session_id") reaches the server.
• The JSON parser on the server side expects valid JSON starting with {, [, or ", but receives incomplete or malformed data instead.
• This is a distributed systems synchronization problem where clients and servers become out of sync during restarts or connection interruptions, resulting in partial or corrupted data transmission.

Resolution

Workaround:

Restart the CLI shell server service to reset the connection state and allow proper client-server synchronization.

1. Log in to the Avi Controller via SSH.
2. Switch to root or sudo mode:
   

   sudo -i

    

3. Restart the CLI shell server service:
   

    systemctl restart clishell_server.service

4. Wait approximately 10-15 seconds for the service to fully initialize.
5. Verify the service is running:
   

   systemctl status clishell_server.service

6. Retry the CLI shell connection.
   
   
   

Alternative Method (Manual Launch):

1. If restarting the service does not resolve the issue, manually launch the shell server as suggested in the error message:
2. Log in to the controller via SSH.
3. Execute the following command:
   

   admin@Controller-IP:~$ sudo -i
   root@Controller-IP:~# python3 /opt/avi/python/bin/cli/bin/shell.py
   Login: admin
   Password:
   [admin:Controller-IP]: >

4. This manually starts the CLI shell server and enables remote connections.

 

Notes: 

• This is a known timing and synchronization issue with the CLI shell server's Python socket implementation.
• The error is typically temporary and resolves after restarting the service.
• The 10-second initialization period is normal behavior and not indicative of a problem.