Issue Too many connections (Several plesk servers same problem in the last week)

[SalvadorS]

Server operating system version

Debian 12.5

Plesk version and microupdate number

18.0.61 #5

Hello,

Last week I had different 5 plesk servers with the same problem. I see the error "too many connections" in the websites or control panel. All the servers are with Debian 12 and Plesk 18.0.61 #5

Restart mysql does not work. If I execute:

sudo service mysql restart || service mariadb restart || service mysqld restart

I have to wait nearly 30 minutes and after that mysql is stopped but not started.

Job for mariadb.service failed because of unavailable resources or another system error.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.

root@server:/home/usr# journalctl -xeu mariadb.service
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599102 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599101 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599100 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599099 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599098 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599097 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599096 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599095 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599094 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599093 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599092 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599091 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599089 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599088 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599086 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599085 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599081 (user : 'hedKrwp75052619') did not exit
jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599080 (user : 'hedKrwp75052619') did not exit
jun 16 22:20:18 dv23 systemd[1]: mariadb.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
jun 16 22:35:18 dv23 systemd[1]: mariadb.service: State 'final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
jun 16 22:35:18 dv23 systemd[1]: mariadb.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'timeout'.
jun 16 22:35:18 dv23 systemd[1]: mariadb.service: Unit process 930 (mariadbd) remains running after unit stopped.
jun 16 22:35:18 dv23 systemd[1]: Stopped mariadb.service - MariaDB 10.11.6 database server.
░░ Subject: A stop job for unit mariadb.service has finished
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A stop job for unit mariadb.service has finished.
░░
░░ The job identifier is 1083555 and the job result is done.
jun 16 22:35:18 dv23 systemd[1]: mariadb.service: Consumed 2d 20h 25min 25.944s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit mariadb.service completed and consumed the indicated resources.
jun 16 22:35:57 dv23 systemd[1]: mariadb.service: Found left-over process 930 (mariadbd) in control group while starting unit. Ignoring.
jun 16 22:35:57 dv23 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
jun 16 22:35:57 dv23 systemd[1]: mariadb.service: Will not start SendSIGKILL=no service of type KillMode=control-group or mixed while processes exist
jun 16 22:35:57 dv23 systemd[1]: mariadb.service: Failed to run 'start-pre' task: Device or resource busy
jun 16 22:35:57 dv23 systemd[1]: mariadb.service: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'resources'.
jun 16 22:35:57 dv23 systemd[1]: mariadb.service: Unit process 930 (mariadbd) remains running after unit stopped.
jun 16 22:35:57 dv23 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server.
░░ Subject: A start job for unit mariadb.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit mariadb.service has finished with a failure.
░░
░░ The job identifier is 1083785 and the job result is failed.

And I have to restart the server to fix the problem. If I restart the server instead of restart mysql I have to wait 30 minutes also.

There are 5 different servers this week with exactly the same problem so I think this could be a plesk problem. Before I open a bug ticket I want to know if someone is facing the same problem or if you know how to prevent it.

Thank you.

 

[SalvadorS]

I found this in the syslog (houndred of lines)

Jun 16 13:31:27 server mariadbd[930]: 2024-06-16 13:31:27 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:27 server mariadbd[930]: 2024-06-16 13:31:27 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:28 server mariadbd[930]: 2024-06-16 13:31:28 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:28 server mariadbd[930]: 2024-06-16 13:31:28 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:31 server mariadbd[930]: 2024-06-16 13:31:31 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:31 server mariadbd[930]: 2024-06-16 13:31:31 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:36 server mariadbd[930]: 2024-06-16 13:31:36 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:36 server mariadbd[930]: 2024-06-16 13:31:36 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:36 server mariadbd[930]: 2024-06-16 13:31:36 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:37 server mariadbd[930]: 2024-06-16 13:31:37 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
Jun 16 13:31:45 server mariadbd[930]: 2024-06-16 13:31:45 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)

 

[SalvadorS]

Also I want to say that in this line:

jun 16 22:05:38 dv23 mariadbd[930]: 2024-06-16 22:05:38 0 [Warning] /usr/sbin/mariadbd: Thread 2599102 (user : 'hedKrwp75052619') did not exit

The user exist, is the only user that exist because the server has only obne site, a wp site with that user on the db...

I nearly convinced that is a plesk problem, nobody have the same issue?

 

[Kaspar@Plesk]

The [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections) error indicate that the maximum number of simultaneous connections to the MySQL databases have been reached and new connections to the server cannot be established.

This would be extremely rare if the server only has one website running. It might indicate a problem or a bug in the website (bad plugin for example). I recommend investigating why there are so many database connections openend in the first place.

Any way, there is an support article that discribes how to increase the maximum number of simultaneous database connections:

https://support.plesk.com/hc/en-us/articles/12377141070999-How-to-increase-the-MySQL-MariaDB-max-connections-value

 

[SalvadorS]

The [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections) error indicate that the maximum number of simultaneous connections to the MySQL databases have been reached and new connections to the server cannot be established.

This would be extremely rare if the server only has one website running. It might indicate a problem a problem or bug the website (bad plugin for example).

Any way, there is an support article that discribes how to increase the maximum number of simultaneous database connections:

https://support.plesk.com/hc/en-us/articles/12377141070999-How-to-increase-the-MySQL-MariaDB-max-connections-value

Dear Dear Kaspar@Plesk,

First of all thanks a lot for your reply. I found this issue in 5 different plesk servers this week. 3 of them with less than 5 sites. Randomly, different hours, different days but same problem. Exactly the same problem. All servers are fresh (mounted few months ago) and the sites are migrated (plesk migration tool) to the new server. All with same software versions. So I think the problem is not increase database connections, I think the problem is another but not in my side. I don´t know if it is mariadb version, plesk version, OS version...

I went to tools and settings -> Diagnostic and repear and check plesk db and mysql -> no problem detected.

So I don´t know what to do...

 

[Bitpalast]

@SalvadorS It's just as @Kaspar@Plesk said. Is "hedKrwp75052619" a user that is your real own database user? Or is is just any attempt to guess at a user name? You could limit the number of instances that a single user can open by adding this to /etc/my.cnf (and restarting MariaDB afterwards):
max_user_connections=n
with n being a reasonable integer value such als 100.

If "hedKrwp75052619" is a database user that your software is using, it probably creates perpetual connections, meaning that it connects but never disconnects from the database, so that each new instance of a script adds a new database session. This is a programming mistake that would need to be fixed in the scripts that you are using.

 

[SalvadorS]

Hi @Bitpalast

First of all, many thanks for your reponse. In this server there is one domain with one wordpress with one database and one username. The username of this database is hedKrwp75052619.

This site was working perfectly for years (I know plugins and corer are updated...)

The only thing I did in all these servers with the same problem is activate the plesk firewall (two weeks ago) but I don´t find any link in these two things... Also is not the only server I have the problem.

Another question, it is possible to limit the number of connections from one user? I know the user (in this case) if I can limit the number of connections I can save the server from a reboot...

Thank you

 

[Hangover2]

Hello @SalvadorS ,

it could also be a "bad crawler" that brings your site down. We had this a lot recently too even though we are using a very good firewall to block most of such bad networks where the crawlers coming from.

Just check your access log of the domain affected with the following command to see, if any IPs are standing out (like several hundred thousand requests):

awk '{ print $1 }' access_ssl_log | sort -n | uniq -c | sort -nr | head -20

Depending on your setup choose the correct logfile from the time, when the problems did occur.

We had "attacks" of bad crawlers with 24 till 32 concurrent connections. This can lead to the problems mentioned if the server is not strong enough and/or there is not good firewall, that could stop those attacks (rate limitation / block bad networks / [...]).

 

[SalvadorS]

Hello @Hangover2

Thanks a lot for your reponse. With fail2ban we blocked in the last month hundred and hundreds of IPs from IA training (claudebot, Amazonbot, Applebot...)

It seems logs are clean of that but if I face the same issue another time I will check...