• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Issue Server is permanently down and unavailable

Jan Bludau said:
I'm not aware of any.
you can only change the plesk bin php-handler otherwise there is no option.

are there that many domains?
Click to expand...

no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

Code: 
ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?
 
fab5freddie said:
but i have also had average values of 600MB per fpm process, so that would be a pm.max_children of 54
Click to expand...
the default for pm.max_children is 100.
fab5freddie said:
no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

Code: 
ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?
Click to expand...

I like to use this shell script

Bash: 
ps -ylC php-fpm --sort:rss | awk '{sum+=$8; ++n} END {print "Total="sum/1024"MB("n")"; print "Average="sum"/"n"="sum/n/1024"MB"}';

really 3200 MB so much, sure that this isn't a memory leak in your script??
Ah you meant in total , okay thats okay.

in your php-fpm settings you should add the following lines

PHP: 
process_control_timeout = 10
request_terminate_timeout = 60

emergency_restart_interval = 1m
emergency_restart_threshold = 3

PHP: Configuration - Manual
emergency_restart_threshold int
If this number of child processes exit with SIGSEGV or SIGBUS within the time interval set by emergency_restart_interval, then FPM will restart. A value of 0 means 'Off'. Default value: 0 (Off).

emergency_restart_interval mixed
Interval of time used by emergency_restart_interval to determine when a graceful restart will be initiated. This can be useful to work around accidental corruptions in an accelerator's shared memory. Available Units: s(econds), m(inutes), h(ours), or d(ays). Default Unit: seconds. Default value: 0 (Off).
Click to expand...
 
fab5freddie said:
no, there aren't that many domains, maybe around 20. but it takes ages for a page to open in the backend, so it would have been easier and above all faster using the sync function.

look, i found this php-fpm calculator:

with which you can calculate exactly all the required data.

pm.max_children = total RAM of the server / average "child process" size, which can be displayed with the following command:

Code: 
ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024, "M") }'

so in my case:
pm.max_children = 32000 MB (RAM) / 117 MB (average php-fpm process) = 273

but what should i set for "reserved ram (gb)" and "ram buffer (%)"?
Click to expand...

Nice PHP-FPM Calculator

How much RAM is your database engine using?

For MariaDB
Bash: 
systemctl status mariadb

For PHP-FPM
Bash: 
systemctl status plesk-php82-fpm
 
Jan Bludau said:
did you change the right mariadb.conf ?

Code: 
nano /etc/mysql/mariadb.conf.d/50-server.cnf
Click to expand...

this code was written for me by chatgpt after i gave it all the data about my server like hardware requirements, databse, plesk version, wordpress instances etc.

Code: 
cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
bind-address = ::ffff:127.0.0.1
local-infile=0

# WordPress-spezifische Optimierungen
innodb_buffer_pool_size=16G  # Erhöht aufgrund des wachsenden Datenbestands
query_cache_size=0  # Query Cache deaktiviert für MariaDB 10.3
max_connections=500  # Weiter erhöht aufgrund des wachsenden Datenbestands

# InnoDB-Einstellungen
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_log_file_size=1G  # Erhöht für eine wachsende Datenbank
innodb_autoinc_lock_mode=2

# Logging
log-error=/var/log/mariadb/mariadb.log
slow_query_log=1
slow_query_log_file=/var/log/mariadb/slow.log

# Sicherheitseinstellungen
skip_name_resolve=1

# Optimierungen für mehrere Kerne
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=4000
innodb_io_capacity_max=8000
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
sql_mode=ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
bind-address = ::ffff:127.0.0.1
local-infile=0

# WordPress-spezifische Optimierungen
innodb_buffer_pool_size=16G  # Erhöht aufgrund des wachsenden Datenbestands
query_cache_size=0  # Query Cache deaktiviert für MariaDB 10.3
max_connections=500  # Weiter erhöht aufgrund des wachsenden Datenbestands

# InnoDB-Einstellungen
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_log_file_size=1G  # Erhöht für eine wachsende Datenbank
innodb_autoinc_lock_mode=2

# Logging
log-error=/var/log/mariadb/mariadb.log
slow_query_log=1
slow_query_log_file=/var/log/mariadb/slow.log

# Sicherheitseinstellungen
skip_name_resolve=1

# Optimierungen für mehrere Kerne
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=4000
innodb_io_capacity_max=8000
 
Jan Bludau said:
this one is more stable because of the usage of swapon -s this exists everywhere @debian.

Code: 
#!/bin/bash
# Display RAM usage in percent
echo "RAM Usage:"
free | awk '/Mem/{printf("Used: %.2f% \n"), $3/$2*100.0}'
# Display Swap usage in percent
echo "Swap Usage:"
SWAP_TOTAL=$(swapon -s | awk 'NR>1{print $3}')
SWAP_USED=$(swapon -s | awk 'NR>1{print $4}')
SWAP_USAGE=$(echo "scale=2; $SWAP_USED/$SWAP_TOTAL*100" | bc)
echo "Used: $SWAP_USAGE%"
# Check if the Swap usage is over 50%
if (( $(echo "$SWAP_USAGE > 50" | bc -l) )); then
    echo 'Swap usage is over 50%, restarting MariaDB...'
    sudo systemctl restart mariadb
else
    echo 'Swap usage is under 50%, no need to restart MariaDB.'
fi
Click to expand...

jan, may i ask you for a favor? could you please extend this script to trigger a "kill" or "restart all php-fpm processes" when the (ram) memory reaches 90%?

thank you very much in advance!
 
Are you aware that RAM is not your issue (see your own post above where you sum it up to only 22 GB out of 32 GB), but too many long-running requests against websites on your server? Regarding the constant high RAM usage, please first check this to understand what Linux does with RAM: Help! Linux ate my RAM!

Next step would be to check which domains consume the most cpu power. You can determine this by
# ps aux | grep php-fpm
or see a sorted output "live" of the 20 most frequented processes:
# watch "ps aux | sort -nrk 3,3 | head -n 20"

Then go into /var/www/vhosts/<subscription>/logs[/<add-on-domain>]/access_ssl_log of that domain and check what is happening in your website. For example check from which ip addresses the most traffic comes in:
# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10
Are these valid sources or are they bad bots? Maybe blocking the sources can solve your server issues?

Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.
Code: 
# systemctl set-property mariadb MemoryAccounting=true
# systemctl set-property mariadb MemoryMax=12G
# systemctl set-property mariadb CPUAccounting=true
# systemctl set-property mariadb CPUQuota=50%
# systemctl daemon-reload
# systemctl restart mariadb
But please make sure that you understand what these commands are doing before you apply anyof them, because especially percentages are always relative to all other percenttage settings affecting the same service. Maybe this can help:
RHEL7: How to get started with CGroups. - CertDepot
 
Peter Debik said:
Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.
Click to expand...
Are you sure this will help? IMHO limiting the database server memory will only ensure that it definitely fails, which is not really helpful.
 
I am not sure whether it fails. Example: When the database needs temporary tables, it can create them in RAM, but when RAM is insufficient it will create them on disk. Similar situation with the buffers. When not enough RAM is available for the buffer size defined in the my.cnf file, it just won't be able to put more into the buffer, but it will not crash. Or will it?
 
Peter Debik said:
Are you aware that RAM is not your issue (see your own post above where you sum it up to only 22 GB out of 32 GB), but too many long-running requests against websites on your server? Regarding the constant high RAM usage, please first check this to understand what Linux does with RAM: Help! Linux ate my RAM!

Next step would be to check which domains consume the most cpu power. You can determine this by
# ps aux | grep php-fpm
or see a sorted output "live" of the 20 most frequented processes:
# watch "ps aux | sort -nrk 3,3 | head -n 20"

Then go into /var/www/vhosts/<subscription>/logs[/<add-on-domain>]/access_ssl_log of that domain and check what is happening in your website. For example check from which ip addresses the most traffic comes in:
# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10
Are these valid sources or are they bad bots? Maybe blocking the sources can solve your server issues?

Regarding cgroups: Are you using the cgroups extension that comes with the Plesk Web Pro and Plesk Web Host editions for free? If so, maybe limiting the resource usage of the problematic websites can help. If not and if you believe the issue is in the database server, you could limit the resource usage of your database service "mariadb" by creating rules on the operating system level, e.g.
Code: 
# systemctl set-property mariadb MemoryAccounting=true
# systemctl set-property mariadb MemoryMax=12G
# systemctl set-property mariadb CPUAccounting=true
# systemctl set-property mariadb CPUQuota=50%
# systemctl daemon-reload
# systemctl restart mariadb
But please make sure that you understand what these commands are doing before you apply anyof them, because especially percentages are always relative to all other percenttage settings affecting the same service. Maybe this can help:
RHEL7: How to get started with CGroups. - CertDepot
Click to expand...

hi guys,

this server is really driving me crazy. my websites were down for 9 hours today! unfortunately i can't use cgroups because i probably don't have the required plesk edition provided by my server provider.

i have tried the systemctl commands mentioned above and it seems to have helped a bit. i no longer have a load average of 100, but am now at values of 0.xx

but the ram memory is still at almost 100%! it is impossible to access the websites. only when i kill the php-fpm instances via ssh, the websites are accessible. but navigating on the website is also extremely slow.

and i noticed that my swap partition never seems to be used for swapping. how can i find out if my swap partition is active and really working?

does anyone else have any ideas what i can do?

thanks in advance
 

Attachments

  • 2023.11.28.jpg
    2023.11.28.jpg
    22.3 KB · Views: 7
  • 2023.11.28_2.jpg
    2023.11.28_2.jpg
    22.3 KB · Views: 8
  • photo_2023-11-28_18-41-36.jpg
    photo_2023-11-28_18-41-36.jpg
    66.3 KB · Views: 8
fab5freddie said:
i can't use cgroups because i probably don't have the required plesk edition provided by my server provider.
Click to expand...
That is why I mentioned the operating system commands. If Cgroups is available on the operating system, you can use it, but have to configure it through command line commands. Did you see the article that I quoted? I will do my best to provide you with suitable information. It would be nice if this did not go to waste.

fab5freddie said:
i have tried the systemctl commands mentioned above and it seems to have helped a bit. i no longer have a load average of 100, but am now at values of 0.xx
Click to expand...
Good.

fab5freddie said:
but the ram memory is still at almost 100%!
Click to expand...
Which RAM is at 100%? You do understand that Linux always utilizes all RAM, right? Did you see the "Linux ate my RAM" article?

fab5freddie said:
t is impossible to access the websites. only when i kill the php-fpm instances via ssh, the websites are accessible. but navigating on the website is also extremely slow.
Click to expand...
That is because the issue is with traffic hitting the websites, not with the database. Did you check it as described in my post above?

fab5freddie said:
and i noticed that my swap partition never seems to be used for swapping. how can i find out if my swap partition is active and really working?
Click to expand...
It's not being used, because you do not have a RAM issue.
 
Peter Debik said:
Did you see the article that I quoted?
Click to expand...
yes, i understood that the ram consumption under linux is normal.

Peter Debik said:
# awk '{ print $1}' access_ssl_log | sort | uniq -c | sort -nr | head -n 10
Click to expand...
i received the following ip addresses in response to this command:
218 208.115.199.19
187 34.196.51.17
148 192.0.91.230
65 89.22.122.116
16 95.235.84.199
9 192.0.103.4
6 35.157.144.199
6 217.138.196.105
5 3.15.153.53
4 65.154.226.171

Peter Debik said:
Which RAM is at 100%? You do understand that Linux always utilizes all RAM, right? Did you see the "Linux ate my RAM" article?
Click to expand...
yes, i understood that. i have understood that my problem is not the ram but the many php-fpm processes.

Peter Debik said:
That is because the issue is with traffic hitting the websites, not with the database. Did you check it as described in my post above?
Click to expand...
yes, i have

Peter Debik said:
It's not being used, because you do not have a RAM issue.
Click to expand...
OK, got it. but what can i do to reduce the traffic/php-fpm processes?
 
fab5freddie said:
OK, got it. but what can i do to reduce the traffic/php-fpm processes?
Click to expand...

I suggest you should check for bots in the access_ssl_log files:
# grep -Ri bot /var/www/vhosts/system/*/logs/access_ssl_log | grep -v robots.txt | wc -l

Identify the bots and block them in the nginx config for each subscription, like this:

Issue - High CPU usage and Swap full despite have RAM avalible

Good morning, as I say in the title we have large CPU usage of our hosts, the hosts that have more traffic are Prestashop with version 1.7.8.8.8 with PHP version 7.4.33, for these hosts we use PHP FPM for Apache and Nginx as a proxy. In addition to this we have an excessive use of SWAP, I attach...
talk.plesk.com talk.plesk.com
 
Maarten. said:
I suggest you should check for bots in the access_ssl_log files:
# grep -Ri bot /var/www/vhosts/system/*/logs/access_ssl_log | grep -v robots.txt | wc -l

Identify the bots and block them in the nginx config for each subscription, like this:

Issue - High CPU usage and Swap full despite have RAM avalible

Good morning, as I say in the title we have large CPU usage of our hosts, the hosts that have more traffic are Prestashop with version 1.7.8.8.8 with PHP version 7.4.33, for these hosts we use PHP FPM for Apache and Nginx as a proxy. In addition to this we have an excessive use of SWAP, I attach...
talk.plesk.com talk.plesk.com
Click to expand...
is there a way to implement fail2ban or an extension that accesses a database that is kept up2date by someone so that the bots are automatically blacklisted on the server?
 
You must log in or register to reply here.

Similar threads

A
Issue Server down: user_beancounters - physpages problem?
Replies
5
Views
207
trialotto
T
M
Resolved No return or error displayed with phpinfo()
Replies
4
Views
424
MatthieuBL
M
E
Issue Abnormal CPU usage and Apache crash
Replies
13
Views
932
MartinT
M
Lrnt
Resolved curl: (6) Could not resolve host Unable to fetch URL only sometimes
Replies
11
Views
1K
AYamshanov
AYamshanov
hansitheking
Resolved Let's Encrypt: Timeout during connect (likely firewall problem)
Replies
4
Views
5K
trialotto
T
Back
Top