ERROR Bad magic header in tc log

Products

• VMware Integrated OpenStack

Issue/Introduction

• Environment is stuck in starting state
• MariaDB will not start
• You may be trying to restart MariaDB and see the following error:
  Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b'2025-02-04 15:29:58 139854286731200 [ERROR] Bad magic header in tc log'
Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b"2025-02-04 15:29:58 139854286731200 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}"
Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b"2025-02-04 15:29:58 139854286731200 [ERROR] Can't init tc log"
Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b'2025-02-04 15:29:58 139854286731200 [ERROR] Aborting'
Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b''
Feb 04 15:34:53 controller-########## mariadb[761]: 2025-02-04 15:34:53,420 - OpenStack-Helm Mariadb - INFO - b"Error in my_thread_global_end(): 1 threads didn't exit"

Environment

7.x

Cause

This might mean that the transaction coordinator log is corrupt.

Resolution

We need to remove the tc.log.

1. ssh to vio-manager
2. osctl edit sts mariadb-server
3. Change  (Indentation matters, use space instead of tabs)
           containers:
           - command:
             - /tmp/start.py
   to
           containers:
           - command:
             - sleep
             - infinity
4. Save the change and exit
5. Wait until the new mariadb-server-0 pod is created
6. osctl exec -ti mariadb-server-0 bash
7. mv /var/lib/mysql/tc.log /tmp
8. revert change to stateful set from above (step 3)

You should see that the mariadb pods will start up without error.  The rest of the services/pods should now start up.