• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Issue Server is permanently down and unavailable

F

fab5freddie

Basic Pleskian
Server operating system version
AlmaLinux 8.9
Plesk version and microupdate number
18.0.56 #4
hi everyone,

i've been trying for days to get my server stable again, but unfortunately i can't get it to work. i've followed some of the advice in the forum here, but unfortunately it hasn't helped in my case.

i have a server with:
  • AlmaLinux 8.9 (Midnight Oncilla)
  • mysqladmin Ver 9.1 Distrib 10.3.39-MariaDB, for Linux on x86_64
  • Plesk Obsidian 18.0.56 Update #4
  • 8 cores @ 2.50GHz
  • 32 GB RAM
  • 1TB SSD, of which only 350GB are in use
  • 1GB swap
  • 14 Wordpress sites, all running PHP 8.2

my RAM is permanently 95% utilized. apparently i have too many processes open and msysql is also eating up a lot of it.

i often have 502 or 504 errors, sometimes i can't even log in to plesk and get this 500 exception error:
(ERROR: Plesk\Exception\Database: DB query failed: SQLSTATE[HY000] [2002] Connection refused (Mysql.php:79)).

sometimes the websites cannot establish a connection to their databases, so that I often have no choice but to restart the server several times a day.

maybe someone can help me? i have also looked at the logfiles. The log excerpts show several error messages in connection with the attempt to access the resource "/usr/share/html/ping". It looks like requests to a specific endpoint on my server are failing.

the load average is also always extremely high.

what else can I do?

many thanks in advance

kind regards
freddie
 

Attachments

  • 123Unbenannt.jpg
    123Unbenannt.jpg
    44 KB · Views: 13
can anyone tell me which command is executed with plesk's repair kit to clear the ram memory?

i haven't found a command for this. i would like to create a cron for this command so that the memory is cleaned up regularly until i find a solution for my problem.

thanks
 
Maybe you need a shell-script

Code: 
#!/bin/bash

# Check if the MariaDB service is running
if systemctl is-active --quiet mariadb
then
    echo "MariaDB is running."
else
    echo "MariaDB is not running. Restarting MariaDB..."
    # Restart MariaDB
    systemctl start mariadb
    if systemctl is-active --quiet mariadb
    then
        echo "MariaDB has been successfully restarted."
    else
        echo "Error restarting MariaDB."
    fi
fi

Source: Server: Prozesse automatisch Neustarten (Shell-Script) - BLUDAU IT SERVICES

or better this one:

Code: 
#!/bin/bash

# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Display Swap usage in percent
echo "Swap Usage:"
free | awk '/Swap/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Determine Swap usage in percent
SWAP_USAGE=$(free | awk '/Swap/{printf("%.0f"), $3/$2*100})

# Check if the Swap usage is over 50%
if (( SWAP_USAGE > 50 )); then
    echo "Swap usage is over 50%, restarting MariaDB..."
    sudo systemctl restart mariadb
else
    echo "Swap usage is under 50%, no need to restart MariaDB."
fi

the second one is better for you.

you need to add it to your contab -e

*/5 * * * * bash ~/ram_verbrauch.sh (for example every 5 Minutes a check if the swap file is over 50%)
Source: Shell-Script: Auf RAM und Swap Verbrauch reagieren - BLUDAU IT SERVICES
 
Jan Bludau said:
Maybe you need a shell-script

Code: 
#!/bin/bash

# Check if the MariaDB service is running
if systemctl is-active --quiet mariadb
then
    echo "MariaDB is running."
else
    echo "MariaDB is not running. Restarting MariaDB..."
    # Restart MariaDB
    systemctl start mariadb
    if systemctl is-active --quiet mariadb
    then
        echo "MariaDB has been successfully restarted."
    else
        echo "Error restarting MariaDB."
    fi
fi

Source: Server: Prozesse automatisch Neustarten (Shell-Script) - BLUDAU IT SERVICES

or better this one:

Code: 
#!/bin/bash

# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Display Swap usage in percent
echo "Swap Usage:"
free | awk '/Swap/{printf("Used: %.2f% \n"), $3/$2*100.0}'

# Determine Swap usage in percent
SWAP_USAGE=$(free | awk '/Swap/{printf("%.0f"), $3/$2*100})

# Check if the Swap usage is over 50%
if (( SWAP_USAGE > 50 )); then
    echo "Swap usage is over 50%, restarting MariaDB..."
    sudo systemctl restart mariadb
else
    echo "Swap usage is under 50%, no need to restart MariaDB."
fi

the second one is better for you.

you need to add it to your contab -e

*/5 * * * * bash ~/ram_verbrauch.sh (for example every 5 Minutes a check if the swap file is over 50%)
Source: Shell-Script: Auf RAM und Swap Verbrauch reagieren - BLUDAU IT SERVICES
Click to expand...

thank you very much for your scripts. i've used the second one.
that's the output of your script:

Code: 
RAM Usage:
Used: 59.13%
Swap Usage:
Used: 0.00%
./checkSwapRAM.sh: line 12: unexpected EOF while looking for matching `''
./checkSwapRAM.sh: line 21: syntax error: unexpected end of file
Click to expand...
 
Maarten. said:
Can you show us the output of the ps_mem tool?
Code: 
# dnf install ps_mem
# ps-mem

It's a well-known script:
github.com

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem
github.com github.com
Click to expand...

thanks!
i get an error

Code: 
# dnf install ps_mem
error: rpmdb: BDB0113 Thread/process 1671/140199693011840 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed
 
fab5freddie said:
thank you very much for your scripts. i've used the second one.
that's the output of your script:
Click to expand...

this one is more stable because of the usage of swapon -s this exists everywhere @debian.

Code: 
#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:" 
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:" 
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi


Code: 
root@mail:~# bash ./ram_verbrauch_english.sh
RAM Usage:
Used: 78.16%
Swap Usage:
Used: 11.00%
Swap usage is under 50%, no need to restart MariaDB.

This is working with @Debian 12 maybe you need to install bc "apt-get install bc"

please tell me if this is working :)
 
Jan Bludau said:
this one is more stable because of the usage of swapon -s this exists everywhere @debian.

Code: 
#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:"
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi


Code: 
root@mail:~# bash ./ram_verbrauch_english.sh
RAM Usage:
Used: 78.16%
Swap Usage:
Used: 11.00%
Swap usage is under 50%, no need to restart MariaDB.

This is working with @Debian 12 maybe you need to install bc "apt-get install bc"

please tell me if this is working :)
Click to expand...

thanks! ... this one worked!

./checkSwapRAM.sh
RAM Usage:
Used: 59.66%
Swap Usage:
Used: 0%
Swap usage is under 50%, no need to restart MariaDB.
 
fab5freddie said:
hi everyone,

i've been trying for days to get my server stable again, but unfortunately i can't get it to work. i've followed some of the advice in the forum here, but unfortunately it hasn't helped in my case.

i have a server with:
  • AlmaLinux 8.9 (Midnight Oncilla)
  • mysqladmin Ver 9.1 Distrib 10.3.39-MariaDB, for Linux on x86_64
  • Plesk Obsidian 18.0.56 Update #4
  • 8 cores @ 2.50GHz
  • 32 GB RAM
  • 1TB SSD, of which only 350GB are in use
  • 1GB swap
  • 14 Wordpress sites, all running PHP 8.2

my RAM is permanently 95% utilized. apparently i have too many processes open and msysql is also eating up a lot of it.

i often have 502 or 504 errors, sometimes i can't even log in to plesk and get this 500 exception error:
(ERROR: Plesk\Exception\Database: DB query failed: SQLSTATE[HY000] [2002] Connection refused (Mysql.php:79)).

sometimes the websites cannot establish a connection to their databases, so that I often have no choice but to restart the server several times a day.

maybe someone can help me? i have also looked at the logfiles. The log excerpts show several error messages in connection with the attempt to access the resource "/usr/share/html/ping". It looks like requests to a specific endpoint on my server are failing.

the load average is also always extremely high.

what else can I do?

many thanks in advance

kind regards
freddie
Click to expand...


Be careful with MariaDB 80% memory usage of your RAM.
Especially in the midday hours, more traffic can occur on your server and more PHP "workers" can be started, which use up additional RAM.
If the RAM is less than 1 GB, the swap size should be at least the size of the RAM and at most twice the size of the RAM.
If the RAM is more than 1 GB, the swap size should be at least equal to the square root of the RAM size and at most twice the size of the RAM.
An old rule of thumb states that the swap size should be twice the size of the RAM or 1.5 times the size of the RAM.

Maybe you need to resize your Swapfile look at this article https://www.linuxtechi.com/extend-swap-space-using-swap-file-in-linux/
 
Maarten. said:
Can you show us the output of the ps_mem tool?
Code: 
# dnf install ps_mem
# ps-mem

It's a well-known script:
github.com

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem
github.com github.com
Click to expand...

I got it to work! these are the results of ps_mem

Code: 
ps_mem
 Private  +   Shared  =  RAM used       Program

252.0 KiB +  82.0 KiB = 334.0 KiB       agetty (2)
248.0 KiB + 155.0 KiB = 403.0 KiB       sh
412.0 KiB +  46.0 KiB = 458.0 KiB       log
640.0 KiB +  69.0 KiB = 709.0 KiB       xinetd
604.0 KiB + 111.0 KiB = 715.0 KiB       dovecot
 56.0 KiB + 828.0 KiB = 884.0 KiB       saslauthd (2)
852.0 KiB + 129.0 KiB = 981.0 KiB       psa-pc-remote
976.0 KiB + 102.5 KiB =   1.1 MiB       plesk-ssh-terminal.x86-64
980.0 KiB + 222.0 KiB =   1.2 MiB       systemctl
  1.1 MiB + 116.5 KiB =   1.2 MiB       master
904.0 KiB + 326.0 KiB =   1.2 MiB       dbus-daemon
992.0 KiB + 245.0 KiB =   1.2 MiB       spawn
  1.1 MiB + 160.0 KiB =   1.2 MiB       plesk_saslauthd
  1.1 MiB + 185.5 KiB =   1.3 MiB       pickup
  1.0 MiB + 267.5 KiB =   1.3 MiB       systemd-logind
  1.2 MiB + 196.5 KiB =   1.4 MiB       qmgr
  1.1 MiB + 297.0 KiB =   1.4 MiB       systemd-udevd
  1.3 MiB + 149.5 KiB =   1.5 MiB       tlsmgr
  1.4 MiB + 227.5 KiB =   1.6 MiB       anvil (2)
  2.3 MiB +   1.3 MiB =   3.6 MiB       crond (3)
  2.5 MiB +   1.1 MiB =   3.7 MiB       (sd-pam) (2)
  3.1 MiB + 611.0 KiB =   3.7 MiB       sw-collectd
  1.5 MiB +   2.6 MiB =   4.0 MiB       sw-cp-serverd (2)
  3.8 MiB +   1.6 MiB =   5.4 MiB       systemd (3)
  5.3 MiB + 663.0 KiB =   5.9 MiB       bash (3)
  6.4 MiB +  22.5 KiB =   6.4 MiB       mandb
  7.5 MiB + 105.5 KiB =   7.6 MiB       config
  2.5 MiB +   5.9 MiB =   8.4 MiB       sshd (7)
  8.6 MiB +   1.4 MiB =   9.9 MiB       systemd-journald
  4.5 MiB +   9.9 MiB =  14.4 MiB       sw-engine-fpm (4)
 14.7 MiB +   9.5 KiB =  14.7 MiB       gpx_plesk-json-backend-datasource_linux_amd64
 14.8 MiB +   1.5 MiB =  16.2 MiB       plesk-task-manager
 19.8 MiB +   2.2 MiB =  22.0 MiB       smtpd (9)
 21.4 MiB +   1.4 MiB =  22.8 MiB       agent360
  8.9 MiB +  15.1 MiB =  24.0 MiB       nginx (3)
  9.3 MiB +  14.9 MiB =  24.2 MiB       httpd (7)
 59.7 MiB +   1.9 MiB =  61.5 MiB       rsyslogd
 67.9 MiB +   4.4 MiB =  72.3 MiB       sw-engine (3)
 73.0 MiB +   5.8 MiB =  78.9 MiB       grafana
  1.6 MiB + 531.9 MiB = 533.5 MiB       drwebd.real (2)
  2.1 GiB +   4.6 MiB =   2.1 GiB       mysqld
 18.3 GiB + 247.9 MiB =  18.6 GiB       php-fpm (346)
---------------------------------
                         21.6 GiB
=================================
 
Jan Bludau said:
Be careful with MariaDB 80% memory usage of your RAM.
Especially in the midday hours, more traffic can occur on your server and more PHP "workers" can be started, which use up additional RAM.
If the RAM is less than 1 GB, the swap size should be at least the size of the RAM and at most twice the size of the RAM.
If the RAM is more than 1 GB, the swap size should be at least equal to the square root of the RAM size and at most twice the size of the RAM.
An old rule of thumb states that the swap size should be twice the size of the RAM or 1.5 times the size of the RAM.

Maybe you need to resize your Swapfile look at this article https://www.linuxtechi.com/extend-swap-space-using-swap-file-in-linux/
Click to expand...

so my swap partion should be 64GB?
 
Maarten. said:
Can you show us the output of the ps_mem tool?
Code: 
# dnf install ps_mem
# ps-mem

It's a well-known script:
github.com

GitHub - pixelb/ps_mem: A utility to accurately report the in core memory usage for a program

A utility to accurately report the in core memory usage for a program - pixelb/ps_mem
github.com github.com
Click to expand...

capacity utilization is now even higher!

Code: 
 ps_mem
 Private  +   Shared  =  RAM used       Program

252.0 KiB +  26.0 KiB = 278.0 KiB       agetty (2)
228.0 KiB + 152.5 KiB = 380.5 KiB       sh
388.0 KiB +  40.0 KiB = 428.0 KiB       log
544.0 KiB + 105.0 KiB = 649.0 KiB       dovecot
580.0 KiB +  69.0 KiB = 649.0 KiB       xinetd
 56.0 KiB + 752.0 KiB = 808.0 KiB       saslauthd (2)
824.0 KiB + 138.0 KiB = 962.0 KiB       psa-pc-remote
976.0 KiB + 102.5 KiB =   1.1 MiB       plesk-ssh-terminal.x86-64
  1.0 MiB + 115.5 KiB =   1.1 MiB       master
828.0 KiB + 353.0 KiB =   1.2 MiB       dbus-daemon
  1.1 MiB + 167.0 KiB =   1.2 MiB       plesk_saslauthd
984.0 KiB + 296.5 KiB =   1.3 MiB       systemd-logind
  1.1 MiB + 152.5 KiB =   1.3 MiB       tlsmgr
  1.1 MiB + 182.5 KiB =   1.3 MiB       pickup
  1.1 MiB + 192.5 KiB =   1.3 MiB       qmgr
  1.0 MiB + 295.0 KiB =   1.3 MiB       systemd-udevd
  1.3 MiB + 219.5 KiB =   1.5 MiB       anvil (2)
  2.2 MiB +  57.0 KiB =   2.3 MiB       htop
  1.8 MiB + 841.5 KiB =   2.6 MiB       systemd-journald
  3.2 MiB +   9.5 KiB =   3.2 MiB       gpx_plesk-json-backend-datasource_linux_amd64
  3.5 MiB +  31.5 KiB =   3.5 MiB       repaird
  2.3 MiB +   1.2 MiB =   3.6 MiB       crond (3)
  3.1 MiB + 573.0 KiB =   3.7 MiB       sw-collectd
  1.5 MiB +   2.4 MiB =   3.9 MiB       sw-cp-serverd (2)
  3.8 MiB +   1.6 MiB =   5.4 MiB       (sd-pam) (3)
  5.2 MiB + 574.0 KiB =   5.8 MiB       bash (3)
  4.8 MiB +   2.0 MiB =   6.8 MiB       systemd (4)
  7.5 MiB +  98.5 KiB =   7.6 MiB       config
  2.5 MiB +   5.9 MiB =   8.4 MiB       sshd (7)
  8.9 MiB +   1.6 MiB =  10.5 MiB       plesk-task-manager
  8.1 MiB +  11.6 MiB =  19.7 MiB       sw-engine-fpm (4)
 20.4 MiB +   1.2 MiB =  21.6 MiB       rsyslogd
 19.8 MiB +   2.1 MiB =  21.9 MiB       smtpd (9)
  9.8 MiB +  13.4 MiB =  23.2 MiB       httpd (7)
  9.0 MiB +  15.1 MiB =  24.1 MiB       nginx (3)
 26.1 MiB +   1.5 MiB =  27.5 MiB       agent360
 31.0 MiB +   2.9 MiB =  33.9 MiB       sw-engine (2)
 72.0 MiB +   5.8 MiB =  77.9 MiB       grafana
160.8 MiB +   2.6 MiB = 163.4 MiB       php
168.0 KiB + 530.8 MiB = 531.0 MiB       drwebd.real (2)
  3.3 GiB +   4.6 MiB =   3.3 GiB       mysqld
 25.8 GiB + 250.9 MiB =  26.1 GiB       php-fpm (355)
---------------------------------
                         30.3 GiB
=================================
 

Attachments

  • 456Unbenannt.jpg
    456Unbenannt.jpg
    22.9 KB · Views: 6
fab5freddie said:
so my swap partion should be 64GB?
Click to expand...

1.5x is enough it should work for you 1 GB is really not enough = 48 GB
be careful with PHP-FPM dynamic.

Never start more servers than you have CPU cores, the maximum you can set is the number of threads, which for you is 8 x 2 = 16.

and setup /etc/ningx/nginx.conf

worker_processes 8;

or use

worker_processes auto;

be carefule with to many opened files at your system look @this How to Fix the ‘Too Many Open Files’ Error in Linux | Windows OS Hub
 
Jan Bludau said:
1.5x is enough it should work for you 1 GB is really not enough = 48 GB
be careful with PHP-FPM dynamic.

Never start more servers than you have CPU cores, the maximum you can set is the number of threads, which for you is 8 x 2 = 16.

and setup /etc/ningx/nginx.conf

worker_processes 8;

or use

worker_processes auto;

be carefule with to many opened files at your system look @this How to Fix the ‘Too Many Open Files’ Error in Linux | Windows OS Hub
Click to expand...

use this for high traffic website:

pm = dynamic
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
 
fab5freddie said:
thanks ...
what should i use for low traffic websites and what should i set for pm.max_children and pm.max_requests in both cases?
Click to expand...
for really small websites use "ondemand"

for small - middle sized traffic dynamic with
pm.start_servers = 2-4
pm.min_spare_server = 1-2
pm.max_spare_servers = 4-6

it depends really how many websites on your hosting machine and how many traffic they have.
But @Maarten. you should monitor what kind of service is using how much memory, most of the time it's mariaDB, if you won't had a own database server only for doing SQL Querying ;-)
 
Jan Bludau said:
for really small websites use "ondemand"

for small - middle sized traffic dynamic with
pm.start_servers = 2-4
pm.min_spare_server = 1-2
pm.max_spare_servers = 4-6

it depends really how many websites on your hosting machine and how many traffic they have.
But @Maarten. you should monitor what kind of service is using how much memory, most of the time it's mariaDB, if you won't had a own database server only for doing SQL Querying ;-)
Click to expand...

is there a way to set the php settings for all domains at once? i tried it via the service plans, but it doesn't seem to synchronize.
 
fab5freddie said:
is there a way to set the php settings for all domains at once? i tried it via the service plans, but it doesn't seem to synchronize.
Click to expand...
it should work via the subscription settings.

Service Plans -> Hosting Plans -> Service Plan Name -> PHP Settings .... PHP-FPM Settings and click on “Update & Sync”.
 
Jan Bludau said:
it should work via the subscription settings.

Service Plans -> Hosting Plans -> Service Plan Name -> PHP Settings .... PHP-FPM Settings and click on “Update & Sync”.
Click to expand...

it should, but unfortunately it doesn‘t. is there a way to repair it or is there a possibility to synchronize it with all domains using a command via shell?
 
fab5freddie said:
it should, but unfortunately it doesn‘t. is there a way to repair it or is there a possibility to synchronize it with all domains using a command via shell?
Click to expand...
I'm not aware of any.
you can only change the plesk bin php-handler otherwise there is no option.

are there that many domains?
 
You must log in or register to reply here.

Similar threads

A
Issue Server down: user_beancounters - physpages problem?
Replies
5
Views
207
trialotto
T
M
Resolved No return or error displayed with phpinfo()
Replies
4
Views
422
MatthieuBL
M
E
Issue Abnormal CPU usage and Apache crash
Replies
13
Views
930
MartinT
M
Lrnt
Resolved curl: (6) Could not resolve host Unable to fetch URL only sometimes
Replies
11
Views
1K
AYamshanov
AYamshanov
hansitheking
Resolved Let's Encrypt: Timeout during connect (likely firewall problem)
Replies
4
Views
5K
trialotto
T
Back
Top