Telco cloud automation Manager service failed to start

Products

• VMware Telco Cloud Automation

Issue/Introduction

Zookeeper service failed to start along with app and web-engine service. TCA-M services are not running.

Environment

2.3 & 3.x

Cause

• Zookeeper service stuck with below log entry:

Jan ## ##:##:## test.example.com zookeeper-start[735329]: ####-##-##  ##:##:##,###[myid:] - ERROR [main:ZooKeeperServerMain@90] - Unexpected exception, exiting abnonormally
Jan ## ##:##:## test.example.com zookeeper-start[735329]: ####-##-##  ##:##:##,### [myid:] - ERROR [main:ServiceUtils@42] - Exiting JVM with code 1

Jan ## ##:##:## test.example.com systemd[1]: zookeeper.service: Main process exited, code=exited, status=1/FAILURE
Jan ## ##:##:## test.example.com.ie systemd[1]: zookeeper.service: Failed with result 'exit-code'.
Jan ## ##:##:## test.example.com systemd[1]: zookeeper.service: Scheduled restart job, restart counter is at 5.
Jan ## ##:##:## test.example.com systemd[1]: Stopped Zookeeper.
Jan ## ##:##:## test.example.com systemd[1]: zookeeper.service: Start request repeated too quickly.
Jan ## ##:##:## test.example.com systemd[1]: zookeeper.service: Failed with result 'exit-code'.
Jan ## ##:##:## test.example.com systemd[1]: Failed to start Zookeeper.

• "/common/zookeeper-db/version-2" folder have corrupt file, at some point when the disk was full. Due to which Zookeeper db unable to write to the file and that makes '0 KB' file.

Resolution

1. SSH to TCA-M
2. Navigate to the folder "/common/zookeeper-db/version-2" and list the files in the folder to see any file with '0 KB' 
3. Move the file which is '0 KB'  to different folder and then restart TCA-M server.
4. This will allow Zookeeper DB to write and allow app and web engine services to start and makes GUI accessible.
5.