Issue Server is permanently down and unavailable

[fab5freddie]

I'm not aware of any.
you can only change the plesk bin php-handler otherwise there is no option.

are there that many domains?

no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

PHP-FPM Process Caluculator

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?

 

[fab5freddie]

but i have also had average values of 600MB per fpm process, so that would be a pm.max_children of 54

 

[Jan Bludau]

but i have also had average values of 600MB per fpm process, so that would be a pm.max_children of 54

the default for pm.max_children is 100.

no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

PHP-FPM Process Caluculator

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?

I like to use this shell script

ps -ylC php-fpm --sort:rss | awk '{sum+=$8; ++n} END {print "Total="sum/1024"MB("n")"; print "Average="sum"/"n"="sum/n/1024"MB"}';

really 3200 MB so much, sure that this isn't a memory leak in your script??
Ah you meant in total , okay thats okay.

in your php-fpm settings you should add the following lines

process_control_timeout = 10
request_terminate_timeout = 60

emergency_restart_interval = 1m
emergency_restart_threshold = 3

PHP: Configuration - Manual

emergency_restart_threshold int
If this number of child processes exit with SIGSEGV or SIGBUS within the time interval set by emergency_restart_interval, then FPM will restart. A value of 0 means 'Off'. Default value: 0 (Off).

emergency_restart_interval mixed
Interval of time used by emergency_restart_interval to determine when a graceful restart will be initiated. This can be useful to work around accidental corruptions in an accelerator's shared memory. Available Units: s(econds), m(inutes), h(ours), or d(ays). Default Unit: seconds. Default value: 0 (Off).

 

[Jan Bludau]

no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

PHP-FPM Process Caluculator

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?

Nice PHP-FPM Calculator

How much RAM is your database engine using?

For MariaDB

systemctl status mariadb

For PHP-FPM

systemctl status plesk-php82-fpm

 

[fab5freddie]

Bash:
systemctl status mariadb

6.9G

Bash:
systemctl status plesk-php82-fpm

20.3G

 

[Jan Bludau]

6.9G


20.3G

6.9G


20.3G

did you change the right mariadb.conf ?

nano /etc/mysql/mariadb.conf.d/50-server.cnf

 

[fab5freddie]

did you change the right mariadb.conf ?

nano /etc/mysql/mariadb.conf.d/50-server.cnf

this code was written for me by chatgpt after i gave it all the data about my server like hardware requirements, databse, plesk version, wordpress instances etc.

cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
bind-address = ::ffff:127.0.0.1
local-infile=0

# WordPress-spezifische Optimierungen
innodb_buffer_pool_size=16G  # Erhöht aufgrund des wachsenden Datenbestands
query_cache_size=0  # Query Cache deaktiviert für MariaDB 10.3
max_connections=500  # Weiter erhöht aufgrund des wachsenden Datenbestands

# InnoDB-Einstellungen
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_log_file_size=1G  # Erhöht für eine wachsende Datenbank
innodb_autoinc_lock_mode=2

# Logging
log-error=/var/log/mariadb/mariadb.log
slow_query_log=1
slow_query_log_file=/var/log/mariadb/slow.log

# Sicherheitseinstellungen
skip_name_resolve=1

# Optimierungen für mehrere Kerne
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=4000
innodb_io_capacity_max=8000
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
bind-address = ::ffff:127.0.0.1
local-infile=0

# WordPress-spezifische Optimierungen
innodb_buffer_pool_size=16G  # Erhöht aufgrund des wachsenden Datenbestands
query_cache_size=0  # Query Cache deaktiviert für MariaDB 10.3
max_connections=500  # Weiter erhöht aufgrund des wachsenden Datenbestands

# InnoDB-Einstellungen
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_log_file_size=1G  # Erhöht für eine wachsende Datenbank
innodb_autoinc_lock_mode=2

# Logging
log-error=/var/log/mariadb/mariadb.log
slow_query_log=1
slow_query_log_file=/var/log/mariadb/slow.log

# Sicherheitseinstellungen
skip_name_resolve=1

# Optimierungen für mehrere Kerne
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=4000
innodb_io_capacity_max=8000

 

[fab5freddie]

this one is more stable because of the usage of swapon -s this exists everywhere @debian.

#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:"
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi

jan, may i ask you for a favor? could you please extend this script to trigger a "kill" or "restart all php-fpm processes" when the (ram) memory reaches 90%?

thank you very much in advance!

 

[Peter Debik]

Would it not be a much better solution to limit RAM usage for the service by cgroups?

 

[fab5freddie]

Would it not be a much better solution to limit RAM usage for the service by cgroups?

hi peter, thanks you for your suggestion. could you please give me an example that makes sense for my server requirements?

 

[Peter Debik]

Are you aware that RAM is not your issue (see your own post above where you sum it up to only 22 GB out of 32 GB), but too many long-running requests against websites on your server? Regarding the constant high RAM usage, please first check this to understand what Linux does with RAM: Help! Linux ate my RAM!

Next step would be to check which domains consume the most cpu power. You can determine this by
# ps aux | grep php-fpm
or see a sorted output "live" of the 20 most frequented processes:
# watch "ps aux | sort -nrk 3,3 | head -n 20"

Then go into /var/www/vhosts/<subscription>/logs[/<add-on-domain>]/access_ssl_log of that domain and check what is happening in your website. For example check from which ip addresses the most traffic comes in:
# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10
Are these valid sources or are they bad bots? Maybe blocking the sources can solve your server issues?

Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.

# systemctl set-property mariadb MemoryAccounting=true
# systemctl set-property mariadb MemoryMax=12G
# systemctl set-property mariadb CPUAccounting=true
# systemctl set-property mariadb CPUQuota=50%
# systemctl daemon-reload
# systemctl restart mariadb

But please make sure that you understand what these commands are doing before you apply anyof them, because especially percentages are always relative to all other percenttage settings affecting the same service. Maybe this can help:
RHEL7: How to get started with CGroups. - CertDepot

 

[mow]

Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.

Are you sure this will help? IMHO limiting the database server memory will only ensure that it definitely fails, which is not really helpful.

 

[Peter Debik]

I am not sure whether it fails. Example: When the database needs temporary tables, it can create them in RAM, but when RAM is insufficient it will create them on disk. Similar situation with the buffers. When not enough RAM is available for the buffer size defined in the my.cnf file, it just won't be able to put more into the buffer, but it will not crash. Or will it?

 

[fab5freddie]

Are you aware that RAM is not your issue (see your own post above where you sum it up to only 22 GB out of 32 GB), but too many long-running requests against websites on your server? Regarding the constant high RAM usage, please first check this to understand what Linux does with RAM: Help! Linux ate my RAM!

Next step would be to check which domains consume the most cpu power. You can determine this by
# ps aux | grep php-fpm
or see a sorted output "live" of the 20 most frequented processes:
# watch "ps aux | sort -nrk 3,3 | head -n 20"

Then go into /var/www/vhosts/<subscription>/logs[/<add-on-domain>]/access_ssl_log of that domain and check what is happening in your website. For example check from which ip addresses the most traffic comes in:
# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10
Are these valid sources or are they bad bots? Maybe blocking the sources can solve your server issues?

Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.

# systemctl set-property mariadb MemoryAccounting=true
# systemctl set-property mariadb MemoryMax=12G
# systemctl set-property mariadb CPUAccounting=true
# systemctl set-property mariadb CPUQuota=50%
# systemctl daemon-reload
# systemctl restart mariadb

But please make sure that you understand what these commands are doing before you apply anyof them, because especially percentages are always relative to all other percenttage settings affecting the same service. Maybe this can help:
RHEL7: How to get started with CGroups. - CertDepot

hi guys,

this server is really driving me crazy. my websites were down for 9 hours today! unfortunately i can't use cgroups because i probably don't have the required plesk edition provided by my server provider.

i have tried the systemctl commands mentioned above and it seems to have helped a bit. i no longer have a load average of 100, but am now at values of 0.xx

but the ram memory is still at almost 100%! it is impossible to access the websites. only when i kill the php-fpm instances via ssh, the websites are accessible. but navigating on the website is also extremely slow.

and i noticed that my swap partition never seems to be used for swapping. how can i find out if my swap partition is active and really working?

does anyone else have any ideas what i can do?

thanks in advance

 

[Peter Debik]

i can't use cgroups because i probably don't have the required plesk edition provided by my server provider.

That is why I mentioned the operating system commands. If Cgroups is available on the operating system, you can use it, but have to configure it through command line commands. Did you see the article that I quoted? I will do my best to provide you with suitable information. It would be nice if this did not go to waste.

i have tried the systemctl commands mentioned above and it seems to have helped a bit. i no longer have a load average of 100, but am now at values of 0.xx

Good.

but the ram memory is still at almost 100%!

Which RAM is at 100%? You do understand that Linux always utilizes all RAM, right? Did you see the "Linux ate my RAM" article?

t is impossible to access the websites. only when i kill the php-fpm instances via ssh, the websites are accessible. but navigating on the website is also extremely slow.

That is because the issue is with traffic hitting the websites, not with the database. Did you check it as described in my post above?

and i noticed that my swap partition never seems to be used for swapping. how can i find out if my swap partition is active and really working?

It's not being used, because you do not have a RAM issue.

 

[Maarten]

and i noticed that my swap partition never seems to be used for swapping. how can i find out if my swap partition is active and really working?

Have a look at this article:

https://www.cyberciti.biz/faq/linux-check-swap-usage-command/

 

[fab5freddie]

Did you see the article that I quoted?

yes, i understood that the ram consumption under linux is normal.

# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10

i received the following ip addresses in response to this command:
218 208.115.199.19
187 34.196.51.17
148 192.0.91.230
65 89.22.122.116
16 95.235.84.199
9 192.0.103.4
6 35.157.144.199
6 217.138.196.105
5 3.15.153.53
4 65.154.226.171

Which RAM is at 100%? You do understand that Linux always utilizes all RAM, right? Did you see the "Linux ate my RAM" article?

yes, i understood that. i have understood that my problem is not the ram but the many php-fpm processes.

That is because the issue is with traffic hitting the websites, not with the database. Did you check it as described in my post above?

yes, i have

It's not being used, because you do not have a RAM issue.

OK, got it. but what can i do to reduce the traffic/php-fpm processes?

 

[Maarten]

OK, got it. but what can i do to reduce the traffic/php-fpm processes?

I suggest you should check for bots in the access_ssl_log files:
# grep -Ri bot /var/www/vhosts/system/*/logs/access_ssl_log | grep -v robots.txt | wc -l

Identify the bots and block them in the nginx config for each subscription, like this:

Issue - High CPU usage and Swap full despite have RAM avalible

Good morning, as I say in the title we have large CPU usage of our hosts, the hosts that have more traffic are Prestashop with version 1.7.8.8.8 with PHP version 7.4.33, for these hosts we use PHP FPM for Apache and Nginx as a proxy. In addition to this we have an excessive use of SWAP, I attach...

talk.plesk.com

 

[fab5freddie]

I suggest you should check for bots in the access_ssl_log files:
# grep -Ri bot /var/www/vhosts/system/*/logs/access_ssl_log | grep -v robots.txt | wc -l

that's the rsult of your command:
8686

i will add your bot block list. i hope it helps!

 

[fab5freddie]

I suggest you should check for bots in the access_ssl_log files:
# grep -Ri bot /var/www/vhosts/system/*/logs/access_ssl_log | grep -v robots.txt | wc -l

Identify the bots and block them in the nginx config for each subscription, like this:

Issue - High CPU usage and Swap full despite have RAM avalible

Good morning, as I say in the title we have large CPU usage of our hosts, the hosts that have more traffic are Prestashop with version 1.7.8.8.8 with PHP version 7.4.33, for these hosts we use PHP FPM for Apache and Nginx as a proxy. In addition to this we have an excessive use of SWAP, I attach...

talk.plesk.com

is there a way to implement fail2ban or an extension that accesses a database that is kept up2date by someone so that the bots are automatically blacklisted on the server?