Issue Server is permanently down and unavailable

[fab5freddie]

Server operating system version

AlmaLinux 8.9

Plesk version and microupdate number

18.0.56 #4

hi everyone,

i've been trying for days to get my server stable again, but unfortunately i can't get it to work. i've followed some of the advice in the forum here, but unfortunately it hasn't helped in my case.

i have a server with:

• AlmaLinux 8.9 (Midnight Oncilla)
• mysqladmin Ver 9.1 Distrib 10.3.39-MariaDB, for Linux on x86_64
• Plesk Obsidian 18.0.56 Update #4
• 8 cores @ 2.50GHz
• 32 GB RAM
• 1TB SSD, of which only 350GB are in use
• 1GB swap
• 14 Wordpress sites, all running PHP 8.2

my RAM is permanently 95% utilized. apparently i have too many processes open and msysql is also eating up a lot of it.

i often have 502 or 504 errors, sometimes i can't even log in to plesk and get this 500 exception error:
(ERROR: Plesk\Exception\Database: DB query failed: SQLSTATE[HY000] [2002] Connection refused (Mysql.php:79)).

sometimes the websites cannot establish a connection to their databases, so that I often have no choice but to restart the server several times a day.

maybe someone can help me? i have also looked at the logfiles. The log excerpts show several error messages in connection with the attempt to access the resource "/usr/share/html/ping". It looks like requests to a specific endpoint on my server are failing.

the load average is also always extremely high.

what else can I do?

many thanks in advance

kind regards
freddie

 

[fab5freddie]

can anyone tell me which command is executed with plesk's repair kit to clear the ram memory?

i haven't found a command for this. i would like to create a cron for this command so that the memory is cleaned up regularly until i find a solution for my problem.

thanks

 

[Jan Bludau]

Maybe you need a shell-script

#!/bin/bash

# Check if the MariaDB service is running
if systemctl is-active --quiet mariadb
then
    echo "MariaDB is running."
else
    echo "MariaDB is not running. Restarting MariaDB..."
    # Restart MariaDB
    systemctl start mariadb
    if systemctl is-active --quiet mariadb
    then
        echo "MariaDB has been successfully restarted."
    else
        echo "Error restarting MariaDB."
    fi
fi

Source: Server: Prozesse automatisch Neustarten (Shell-Script) - BLUDAU IT SERVICES

or better this one:

#!/bin/bash

# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Display Swap usage in percent
echo "Swap Usage:"
free | awk '/Swap/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Determine Swap usage in percent
SWAP_USAGE=$(free | awk '/Swap/{printf("%.0f"), $3/$2*100})

# Check if the Swap usage is over 50%
if (( SWAP_USAGE > 50 )); then
    echo "Swap usage is over 50%, restarting MariaDB..."
    sudo systemctl restart mariadb
else
    echo "Swap usage is under 50%, no need to restart MariaDB."
fi

the second one is better for you.

you need to add it to your contab -e

*/5 * * * * bash ~/ram_verbrauch.sh (for example every 5 Minutes a check if the swap file is over 50%)
Source: Shell-Script: Auf RAM und Swap Verbrauch reagieren - BLUDAU IT SERVICES

 

[fab5freddie]

Maybe you need a shell-script

#!/bin/bash

# Check if the MariaDB service is running
if systemctl is-active --quiet mariadb
then
    echo "MariaDB is running."
else
    echo "MariaDB is not running. Restarting MariaDB..."
    # Restart MariaDB
    systemctl start mariadb
    if systemctl is-active --quiet mariadb
    then
        echo "MariaDB has been successfully restarted."
    else
        echo "Error restarting MariaDB."
    fi
fi

Source: Server: Prozesse automatisch Neustarten (Shell-Script) - BLUDAU IT SERVICES

or better this one:

#!/bin/bash

# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Display Swap usage in percent
echo "Swap Usage:"
free | awk '/Swap/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Determine Swap usage in percent
SWAP_USAGE=$(free | awk '/Swap/{printf("%.0f"), $3/$2*100})

# Check if the Swap usage is over 50%
if (( SWAP_USAGE > 50 )); then
    echo "Swap usage is over 50%, restarting MariaDB..."
    sudo systemctl restart mariadb
else
    echo "Swap usage is under 50%, no need to restart MariaDB."
fi

the second one is better for you.

you need to add it to your contab -e

*/5 * * * * bash ~/ram_verbrauch.sh (for example every 5 Minutes a check if the swap file is over 50%)
Source: Shell-Script: Auf RAM und Swap Verbrauch reagieren - BLUDAU IT SERVICES

thank you very much for your scripts. i've used the second one.
that's the output of your script:

RAM Usage:
Used: 59.13%
Swap Usage:
Used: 0.00%
./checkSwapRAM.sh: line 12: unexpected EOF while looking for matching `''
./checkSwapRAM.sh: line 21: syntax error: unexpected end of file

 

[Maarten]

Can you show us the output of the ps_mem tool?

# dnf install ps_mem
# ps-mem

It's a well-known script:

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem

github.com

 

[fab5freddie]

Can you show us the output of the ps_mem tool?

# dnf install ps_mem
# ps-mem

It's a well-known script:

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem

github.com

thanks!
i get an error

# dnf install ps_mem
error: rpmdb: BDB0113 Thread/process 1671/140199693011840 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

 

[Jan Bludau]

thank you very much for your scripts. i've used the second one.
that's the output of your script:

this one is more stable because of the usage of swapon -s this exists everywhere @debian.

#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:" 
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:" 
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi

root@mail:~# bash ./ram_verbrauch_english.sh
RAM Usage:
Used: 78.16%
Swap Usage:
Used: 11.00%
Swap usage is under 50%, no need to restart MariaDB.

This is working with @Debian 12 maybe you need to install bc "apt-get install bc"

please tell me if this is working

 

[fab5freddie]

this one is more stable because of the usage of swapon -s this exists everywhere @debian.

#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:"
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi

root@mail:~# bash ./ram_verbrauch_english.sh
RAM Usage:
Used: 78.16%
Swap Usage:
Used: 11.00%
Swap usage is under 50%, no need to restart MariaDB.

This is working with @Debian 12 maybe you need to install bc "apt-get install bc"

please tell me if this is working

thanks! ... this one worked!

./checkSwapRAM.sh
RAM Usage:
Used: 59.66%
Swap Usage:
Used: 0%
Swap usage is under 50%, no need to restart MariaDB.

 

[Jan Bludau]

hi everyone,

i've been trying for days to get my server stable again, but unfortunately i can't get it to work. i've followed some of the advice in the forum here, but unfortunately it hasn't helped in my case.

i have a server with:

• AlmaLinux 8.9 (Midnight Oncilla)
• mysqladmin Ver 9.1 Distrib 10.3.39-MariaDB, for Linux on x86_64
• Plesk Obsidian 18.0.56 Update #4
• 8 cores @ 2.50GHz
• 32 GB RAM
• 1TB SSD, of which only 350GB are in use
• 1GB swap
• 14 Wordpress sites, all running PHP 8.2

my RAM is permanently 95% utilized. apparently i have too many processes open and msysql is also eating up a lot of it.

i often have 502 or 504 errors, sometimes i can't even log in to plesk and get this 500 exception error:
(ERROR: Plesk\Exception\Database: DB query failed: SQLSTATE[HY000] [2002] Connection refused (Mysql.php:79)).

sometimes the websites cannot establish a connection to their databases, so that I often have no choice but to restart the server several times a day.

maybe someone can help me? i have also looked at the logfiles. The log excerpts show several error messages in connection with the attempt to access the resource "/usr/share/html/ping". It looks like requests to a specific endpoint on my server are failing.

the load average is also always extremely high.

what else can I do?

many thanks in advance

kind regards
freddie

Be careful with MariaDB 80% memory usage of your RAM.
Especially in the midday hours, more traffic can occur on your server and more PHP "workers" can be started, which use up additional RAM.
If the RAM is less than 1 GB, the swap size should be at least the size of the RAM and at most twice the size of the RAM.
If the RAM is more than 1 GB, the swap size should be at least equal to the square root of the RAM size and at most twice the size of the RAM.
An old rule of thumb states that the swap size should be twice the size of the RAM or 1.5 times the size of the RAM.

Maybe you need to resize your Swapfile look at this article https://www.linuxtechi.com/extend-swap-space-using-swap-file-in-linux/

 

[fab5freddie]

Can you show us the output of the ps_mem tool?

# dnf install ps_mem
# ps-mem

It's a well-known script:

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem

github.com

I got it to work! these are the results of ps_mem

ps_mem
 Private  +   Shared  =  RAM used       Program

252.0 KiB +  82.0 KiB = 334.0 KiB       agetty (2)
248.0 KiB + 155.0 KiB = 403.0 KiB       sh
412.0 KiB +  46.0 KiB = 458.0 KiB       log
640.0 KiB +  69.0 KiB = 709.0 KiB       xinetd
604.0 KiB + 111.0 KiB = 715.0 KiB       dovecot
 56.0 KiB + 828.0 KiB = 884.0 KiB       saslauthd (2)
852.0 KiB + 129.0 KiB = 981.0 KiB       psa-pc-remote
976.0 KiB + 102.5 KiB =   1.1 MiB       plesk-ssh-terminal.x86-64
980.0 KiB + 222.0 KiB =   1.2 MiB       systemctl
  1.1 MiB + 116.5 KiB =   1.2 MiB       master
904.0 KiB + 326.0 KiB =   1.2 MiB       dbus-daemon
992.0 KiB + 245.0 KiB =   1.2 MiB       spawn
  1.1 MiB + 160.0 KiB =   1.2 MiB       plesk_saslauthd
  1.1 MiB + 185.5 KiB =   1.3 MiB       pickup
  1.0 MiB + 267.5 KiB =   1.3 MiB       systemd-logind
  1.2 MiB + 196.5 KiB =   1.4 MiB       qmgr
  1.1 MiB + 297.0 KiB =   1.4 MiB       systemd-udevd
  1.3 MiB + 149.5 KiB =   1.5 MiB       tlsmgr
  1.4 MiB + 227.5 KiB =   1.6 MiB       anvil (2)
  2.3 MiB +   1.3 MiB =   3.6 MiB       crond (3)
  2.5 MiB +   1.1 MiB =   3.7 MiB       (sd-pam) (2)
  3.1 MiB + 611.0 KiB =   3.7 MiB       sw-collectd
  1.5 MiB +   2.6 MiB =   4.0 MiB       sw-cp-serverd (2)
  3.8 MiB +   1.6 MiB =   5.4 MiB       systemd (3)
  5.3 MiB + 663.0 KiB =   5.9 MiB       bash (3)
  6.4 MiB +  22.5 KiB =   6.4 MiB       mandb
  7.5 MiB + 105.5 KiB =   7.6 MiB       config
  2.5 MiB +   5.9 MiB =   8.4 MiB       sshd (7)
  8.6 MiB +   1.4 MiB =   9.9 MiB       systemd-journald
  4.5 MiB +   9.9 MiB =  14.4 MiB       sw-engine-fpm (4)
 14.7 MiB +   9.5 KiB =  14.7 MiB       gpx_plesk-json-backend-datasource_linux_amd64
 14.8 MiB +   1.5 MiB =  16.2 MiB       plesk-task-manager
 19.8 MiB +   2.2 MiB =  22.0 MiB       smtpd (9)
 21.4 MiB +   1.4 MiB =  22.8 MiB       agent360
  8.9 MiB +  15.1 MiB =  24.0 MiB       nginx (3)
  9.3 MiB +  14.9 MiB =  24.2 MiB       httpd (7)
 59.7 MiB +   1.9 MiB =  61.5 MiB       rsyslogd
 67.9 MiB +   4.4 MiB =  72.3 MiB       sw-engine (3)
 73.0 MiB +   5.8 MiB =  78.9 MiB       grafana
  1.6 MiB + 531.9 MiB = 533.5 MiB       drwebd.real (2)
  2.1 GiB +   4.6 MiB =   2.1 GiB       mysqld
 18.3 GiB + 247.9 MiB =  18.6 GiB       php-fpm (346)
---------------------------------
                         21.6 GiB
=================================

 

[fab5freddie]

Be careful with MariaDB 80% memory usage of your RAM.
Especially in the midday hours, more traffic can occur on your server and more PHP "workers" can be started, which use up additional RAM.
If the RAM is less than 1 GB, the swap size should be at least the size of the RAM and at most twice the size of the RAM.
If the RAM is more than 1 GB, the swap size should be at least equal to the square root of the RAM size and at most twice the size of the RAM.
An old rule of thumb states that the swap size should be twice the size of the RAM or 1.5 times the size of the RAM.

Maybe you need to resize your Swapfile look at this article https://www.linuxtechi.com/extend-swap-space-using-swap-file-in-linux/

so my swap partion should be 64GB?

 

[fab5freddie]

Can you show us the output of the ps_mem tool?

# dnf install ps_mem
# ps-mem

It's a well-known script:

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem

github.com

capacity utilization is now even higher!

 ps_mem
 Private  +   Shared  =  RAM used       Program

252.0 KiB +  26.0 KiB = 278.0 KiB       agetty (2)
228.0 KiB + 152.5 KiB = 380.5 KiB       sh
388.0 KiB +  40.0 KiB = 428.0 KiB       log
544.0 KiB + 105.0 KiB = 649.0 KiB       dovecot
580.0 KiB +  69.0 KiB = 649.0 KiB       xinetd
 56.0 KiB + 752.0 KiB = 808.0 KiB       saslauthd (2)
824.0 KiB + 138.0 KiB = 962.0 KiB       psa-pc-remote
976.0 KiB + 102.5 KiB =   1.1 MiB       plesk-ssh-terminal.x86-64
  1.0 MiB + 115.5 KiB =   1.1 MiB       master
828.0 KiB + 353.0 KiB =   1.2 MiB       dbus-daemon
  1.1 MiB + 167.0 KiB =   1.2 MiB       plesk_saslauthd
984.0 KiB + 296.5 KiB =   1.3 MiB       systemd-logind
  1.1 MiB + 152.5 KiB =   1.3 MiB       tlsmgr
  1.1 MiB + 182.5 KiB =   1.3 MiB       pickup
  1.1 MiB + 192.5 KiB =   1.3 MiB       qmgr
  1.0 MiB + 295.0 KiB =   1.3 MiB       systemd-udevd
  1.3 MiB + 219.5 KiB =   1.5 MiB       anvil (2)
  2.2 MiB +  57.0 KiB =   2.3 MiB       htop
  1.8 MiB + 841.5 KiB =   2.6 MiB       systemd-journald
  3.2 MiB +   9.5 KiB =   3.2 MiB       gpx_plesk-json-backend-datasource_linux_amd64
  3.5 MiB +  31.5 KiB =   3.5 MiB       repaird
  2.3 MiB +   1.2 MiB =   3.6 MiB       crond (3)
  3.1 MiB + 573.0 KiB =   3.7 MiB       sw-collectd
  1.5 MiB +   2.4 MiB =   3.9 MiB       sw-cp-serverd (2)
  3.8 MiB +   1.6 MiB =   5.4 MiB       (sd-pam) (3)
  5.2 MiB + 574.0 KiB =   5.8 MiB       bash (3)
  4.8 MiB +   2.0 MiB =   6.8 MiB       systemd (4)
  7.5 MiB +  98.5 KiB =   7.6 MiB       config
  2.5 MiB +   5.9 MiB =   8.4 MiB       sshd (7)
  8.9 MiB +   1.6 MiB =  10.5 MiB       plesk-task-manager
  8.1 MiB +  11.6 MiB =  19.7 MiB       sw-engine-fpm (4)
 20.4 MiB +   1.2 MiB =  21.6 MiB       rsyslogd
 19.8 MiB +   2.1 MiB =  21.9 MiB       smtpd (9)
  9.8 MiB +  13.4 MiB =  23.2 MiB       httpd (7)
  9.0 MiB +  15.1 MiB =  24.1 MiB       nginx (3)
 26.1 MiB +   1.5 MiB =  27.5 MiB       agent360
 31.0 MiB +   2.9 MiB =  33.9 MiB       sw-engine (2)
 72.0 MiB +   5.8 MiB =  77.9 MiB       grafana
160.8 MiB +   2.6 MiB = 163.4 MiB       php
168.0 KiB + 530.8 MiB = 531.0 MiB       drwebd.real (2)
  3.3 GiB +   4.6 MiB =   3.3 GiB       mysqld
 25.8 GiB + 250.9 MiB =  26.1 GiB       php-fpm (355)
---------------------------------
                         30.3 GiB
=================================

 

[Jan Bludau]

so my swap partion should be 64GB?

1.5x is enough it should work for you 1 GB is really not enough = 48 GB
be careful with PHP-FPM dynamic.

Never start more servers than you have CPU cores, the maximum you can set is the number of threads, which for you is 8 x 2 = 16.

and setup /etc/ningx/nginx.conf

worker_processes 8;

or use

worker_processes auto;

be carefule with to many opened files at your system look @this How to Fix the ‘Too Many Open Files’ Error in Linux | Windows OS Hub

 

[Jan Bludau]

1.5x is enough it should work for you 1 GB is really not enough = 48 GB
be careful with PHP-FPM dynamic.

Never start more servers than you have CPU cores, the maximum you can set is the number of threads, which for you is 8 x 2 = 16.

and setup /etc/ningx/nginx.conf

worker_processes 8;

or use

worker_processes auto;

be carefule with to many opened files at your system look @this How to Fix the ‘Too Many Open Files’ Error in Linux | Windows OS Hub

use this for high traffic website:

pm = dynamic
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16

 

[fab5freddie]

use this for high traffic website:

pm = dynamic
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16

thanks ...
what should i use for low traffic websites and what should i set for pm.max_children and pm.max_requests in both cases?

 

[Jan Bludau]

thanks ...
what should i use for low traffic websites and what should i set for pm.max_children and pm.max_requests in both cases?

for really small websites use "ondemand"

for small - middle sized traffic dynamic with
pm.start_servers = 2-4
pm.min_spare_server = 1-2
pm.max_spare_servers = 4-6

it depends really how many websites on your hosting machine and how many traffic they have.
But @Maarten. you should monitor what kind of service is using how much memory, most of the time it's mariaDB, if you won't had a own database server only for doing SQL Querying ;-)

 

[fab5freddie]

for really small websites use "ondemand"

for small - middle sized traffic dynamic with
pm.start_servers = 2-4
pm.min_spare_server = 1-2
pm.max_spare_servers = 4-6

it depends really how many websites on your hosting machine and how many traffic they have.
But @Maarten. you should monitor what kind of service is using how much memory, most of the time it's mariaDB, if you won't had a own database server only for doing SQL Querying ;-)

is there a way to set the php settings for all domains at once? i tried it via the service plans, but it doesn't seem to synchronize.

 

[Jan Bludau]

is there a way to set the php settings for all domains at once? i tried it via the service plans, but it doesn't seem to synchronize.

it should work via the subscription settings.

Service Plans -> Hosting Plans -> Service Plan Name -> PHP Settings .... PHP-FPM Settings and click on “Update & Sync”.

 

[fab5freddie]

it should work via the subscription settings.

Service Plans -> Hosting Plans -> Service Plan Name -> PHP Settings .... PHP-FPM Settings and click on “Update & Sync”.

it should, but unfortunately it doesn‘t. is there a way to repair it or is there a possibility to synchronize it with all domains using a command via shell?

 

[Jan Bludau]

it should, but unfortunately it doesn‘t. is there a way to repair it or is there a possibility to synchronize it with all domains using a command via shell?

I'm not aware of any.
you can only change the plesk bin php-handler otherwise there is no option.

are there that many domains?