SE container continuously crashing in a Linux Server cloud environment

Products

• VMware Avi Load Balancer

Issue/Introduction

- SE container continuously crashing in LSC cloud.
- The SE is stuck in initializing state on the UI.

Environment

- Avi controller version <22.1.7 and <30.2.2
- Linux server cloud
- Docker version >24.x

Cause

- Docker version greater than 24.x support minimum api version as 1.24, but in AVI SE docker image we are setting docker_api_version to 1.12 which is no longer compatible with docker version >24.x and onwards.

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24) 

- During SE container initialization, there is a specific script run which is hardcoding the docker api version to 1.12.
- Because of this, we see the SE container go into a continuous reboot loop. From the SE host, you can see errors like:

[root@localhost ~]# tail -f /var/log/avise.log | zgrep 'Starting the avise container'
[Thu Dec 5 07:12:04 EST 2024] Starting the avise container
[Thu Dec 5 07:12:05 EST 2024] Starting the avise container
[Thu Dec 5 07:12:06 EST 2024] Starting the avise container
[Thu Dec 5 07:12:07 EST 2024] Starting the avise container
[Thu Dec 5 07:12:07 EST 2024] Starting the avise container
[Thu Dec 5 07:12:08 EST 2024] Starting the avise container

- We can also see "shim disconnected" logs in /var/log/messages on the SE host:

[root@localhost ~]# zgrep 'disconnected' /var/log/messages | tail -n 5
Dec  5 07:13:44 localhost containerd: time="2024-12-05T07:13:44.239792720-05:00" level=warning msg="cleaning up after shim disconnected" id=e6e11b727cb12b248e66836912ec3d8cbd32dbacc44282257b9ea203522d37c6 namespace=moby
Dec  5 07:13:46 localhost containerd: time="2024-12-05T07:13:46.805320296-05:00" level=info msg="shim disconnected" id=9ea8a431f5859a35c3e06e9136d165d07f643febd12859ffc3e92aef7f86ad52
Dec  5 07:13:46 localhost containerd: time="2024-12-05T07:13:46.805374924-05:00" level=warning msg="cleaning up after shim disconnected" id=9ea8a431f5859a35c3e06e9136d165d07f643febd12859ffc3e92aef7f86ad52 namespace=moby
Dec  5 07:13:48 localhost containerd: time="2024-12-05T07:13:48.378474582-05:00" level=info msg="shim disconnected" id=2acccb54c95512ac7a6f29ca8a5ffff42e8333639ab1b22eeeb4432c441eab63
Dec  5 07:13:48 localhost containerd: time="2024-12-05T07:13:48.378542723-05:00" level=warning msg="cleaning up after shim disconnected" id=2acccb54c95512ac7a6f29ca8a5ffff42e8333639ab1b22eeeb4432c441eab63 namespace=moby

- NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Resolution

- We have removed this hardcoding of DOCKER_API_VERSION in 22.1.7+, 30.2.2+.
- The recommendation would be to upgrade to the latest version.