kernel: mysqld segfault on cPanel Servers (segmentation fault)

cPanel has recently rolled out an update for its control panel, which includes an upgrade to MySQL version 8.0.38. However, following this update, many users have encountered a critical issue where MySQL ceases to respond, resulting in a segmentation fault (mysqld segfault cpanel). This unexpected problem has disrupted database operations, causing significant concern among cPanel users who rely on the stability and reliability of MySQL for their web hosting and server management tasks.

The error looks like this and it is impossible to get MySQL to run again.

Jul  1 00:36:35 server kernel: [ 3335.191127] boot[100978]: segfault at 508 ip 000055ca2e677670 sp 00007fc870c729c0 error 4 in mysqld[55ca2d0d2000+201b000]

Workaround if your server is broken already with mysqld segfault on cPanel servers

For RHEL-based (RockyLinux, AlmaLinux, CentOS) servers:

To address the mysqld segfault issue, you can downgrade to the previous subversion of MySQL 8. Follow these steps:

1. Open your terminal.
2. Execute the following command to downgrade MySQL:

yum downgrade mysql-community-*

Then you can try restarting mysql by running: service mysql restart

For Ubuntu servers:

Due to the limitations of the apt command, downgrading MySQL packages natively is not possible. Instead, you need to manually downgrade the packages using the MySQL archive.

1. Create a Directory for Downgrade Files:

• Log in as the root user.
• Create a new directory called mysqldown in the root folder and navigate to it:

mkdir -v /root/mysqldown && cd /root/mysqldown

2. Download the MySQL 8.0.37 Archive:

If your operating system is Ubuntu 20.04:

wget https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.37-1ubuntu20.04_amd64.deb-bundle.tar

If your operating system is Ubuntu 22.04:

wget https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.37-1ubuntu22.04_amd64.deb-bundle.tar

3. Extract the MySQL 8.0.37 Archive Bundle:

If your operating system is Ubuntu 20.04:

tar xvf mysql-server_8.0.37-1ubuntu20.04_amd64.deb-bundle.tar

If your operating system is Ubuntu 22.04:

tar xvf mysql-server_8.0.37-1ubuntu22.04_amd64.deb-bundle.tar

4. Remove Unnecessary Packages:

If your operating system is Ubuntu 20.04:

rm -rvf ./mysql-community-test-debug_8.0.37-1ubuntu20.04_amd64.deb ./mysql-community-test_8.0.37-1ubuntu20.04_amd64.deb ./mysql-testsuite_8.0.37-1ubuntu20.04_amd64.deb

If your operating system is Ubuntu 22.04:

rm -rvf ./mysql-community-test-debug_8.0.37-1ubuntu22.04_amd64.deb ./mysql-community-test_8.0.37-1ubuntu22.04_amd64.deb ./mysql-testsuite_8.0.37-1ubuntu22.04_amd64.deb

5. Install the older (working version) of MySQL Packages

dpkg -i *mysql*.deb

6. Verify MySQL is working again

/scripts/restartsrv_mysql --status

7. Prevent MySQL Updates

To prevent these packages from being updated again, use the following command to version-lock them:

apt-mark hold mysql-client mysql-common mysql-community-client-core mysql-community-client-plugins

How to temporarily prevent the mysqld segfault error from happening (if your server is not broken yet)

For RHEL-based (RockyLinux, AlmaLinux, CentOS) servers:

1. Edit /etc/yum.conf

Modify the line which includes: exclude=

Add at the end of the line: mysql-community*

On my server it looks like this (but don’t copy it): exclude=bind-chroot dovecot* exim* filesystem p0f php* proftpd* pure-ftpd* mysql-community*

For Ubuntu servers:

Disable updates using apt-mark by running:

apt-mark hold mysql-client mysql-common mysql-community-client-core mysql-community-client-plugins

After cPanel issues a fix you CAN and you SHOULD remove this so that your server is correctly updated.

More issue about this issue can be found on cPanel’s Support Page Upstream issue: MySQL 8.0.38 does not start (Segmentation Fault)