HTTP ERROR 503 Service Unavailable after upgrading to 22.2.4

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Working on upgrading my test environment from 22.2.3 to 22.2.4. 

DR and perf center console upgrade was going well.

 

Had console open and during sync of DA after its upgrade the GUI crashed.

Now just getting error 503 on the console page.

 

In the mysql error log (/opt/CA/MySql/data/HOSTNAME.err)

We see that the mysql panics during the DA synch and never recovers:

**********

2023-01-19T14:33:01.693819Z 55 [ERROR] [MY-012153] [InnoDB] Trying to access page number 825241649 in space 129957, space name netqosportal/dst_device, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.

2023-01-19T14:33:01.693898Z 55 [ERROR] [MY-012154] [InnoDB] Server exits.

2023-01-19T14:33:01.693914Z 55 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 140315112466176

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

Environment

Release : 22.2

Cause

There was an endpoint agent running on the systems called

Crowdstrike

This tool was interfering with the mysql files.

Resolution

Rolled back to snapshot.

Set the Crowdstrike endpoint agent into “bypass mode”

Run the upgrade successfully.

Additional Information

what-is-crowdstrike