Resolved To many Apache process eating memory

[Cristian Rodriguez]

Since a month ago my dedicated server is experiencing performance problems, it is slow, tring to figure out what the problem is and applying many changes to the server nothing has changed websites are loading very slow, what i have notice is that there are many apache process (httpd) and every process is eating too much memory.

Server information:

70 subscriptions
Disk 1TB (50% used)
Ram 8 GB


top - 20:50:41 up 3 days, 3:05, 1 user, load average: 81.77, 95.44, 86.96
Tasks: 577 total, 1 running, 571 sleeping, 0 stopped, 5 zombie
Cpu(s): 2.5%us, 1.4%sy, 0.0%ni, 39.1%id, 56.4%wa, 0.2%hi, 0.3%si, 0.0%st
Mem: 7932144k total, 4766860k used, 3165284k free, 1448800k buffers
Swap: 16383996k total, 41664k used, 16342332k free, 723568k cached


If Memory is over 50% websites speed start to decrese


25411 apache 20 0 466m 93m 2936 D 0.0 1.2 0:00.06 httpd
25253 apache 20 0 466m 93m 2936 S 0.0 1.2 0:00.07 httpd
25237 apache 20 0 466m 93m 2924 S 0.0 1.2 0:00.07 httpd
25259 apache 20 0 466m 93m 2924 S 0.0 1.2 0:00.07 httpd
25431 apache 20 0 466m 93m 2936 S 0.0 1.2 0:00.06 httpd
25352 apache 20 0 466m 93m 2932 S 0.0 1.2 0:00.07 httpd
25412 apache 20 0 466m 93m 2936 S 0.0 1.2 0:00.06 httpd
26732 apache 20 0 466m 93m 2928 S 0.0 1.2 0:00.05 httpd
25275 apache 20 0 466m 93m 2936 S 0.0 1.2 0:00.06 httpd
25354 apache 20 0 466m 93m 2928 S 0.0 1.2 0:00.06 httpd
25235 apache 20 0 466m 93m 2944 S 0.0 1.2 0:00.06 httpd
28500 apache 20 0 466m 93m 2924 S 0.0 1.2 0:00.04 httpd
28924 apache 20 0 465m 92m 2188 S 0.0 1.2 0:00.01 httpd


Every process may be a child and a child a connection so, if there are 50 process they are eating almos 5GB of ram, so i thing every process is taking too much memory, i think it should be lower.

Some say that keep alive should be Off if there are many connections or ram memory is poor, so i changed this value but noting changed.

I have made many improvements like:

Change cron jobs to server low usage hours.
Disable Wordpress wp-cron.php defail behaivor for a cron job process.
Mitigate Wordpress attack with fail2ban
Reduce and increase MaxClients
Activate NGNIX for all sites


But nothing seems to work, i can not lower ram memory usage, i thing the dedicated server should have a better performance but i does not.

Hardware health is ok.

Another thing i can see and maybe is related but should not affect normally is that there are a lot of connections form Cloudflare, now there is just one website using Cloudflare but there are 95 established connections i think is too much, it is at least the 80% of current connections.

Most of the sites should not have to many connections most of the usage is for the email service through imap or pop3

UPDATE

Digging a little bit i found that part of the problem is disk IO caused by statics collector, some of the task in my case are not need, spamtrain is similar to sa-update that is executed separately, i guess i can run every task separately in a cron job and not to run a task i don't need.

As explained in Plesk Support:

# /usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -h

I think it is a bad idea to join multiple task in one process, by now i killed the process, customers has the priority and they need efficientcy.

there are other process causing high IO but i think it is easier to resolve.

 

[Bitpalast]

A first approach to reduce the Apache memory consumption could be to turn off all Apache modules that are not absolutely needed by your sites. This can be done by unchecking the modules in the Apache web server configuration page in Tools & Settings.

Further, in your output the swap file is indeed used. Normally, swap files should not be used at all and only exist in case the conventional RAM is insufficient. The moment a swap file is used, the overall response time increases dramatically, because now a lot of data needs to be written to and read from disk. The server is not slowing down when 50 % (or even more like for example 90%) of RAM is used, but it becomes slow when all RAM is used up and the system starts swapping.

 

[Cristian Rodriguez]

Thank you for the response

I am going to review what modules are in use and turn them off, i agree with you memory usage should not be the problem, my new approach is that the problem is due to Disk IO causing high load average.

# atop
MDD | md1 | busy 0% | read 1817 | write 949 | KiB/w 3 | MBr/s 2.91 | MBw/s 0.37 | avio 0.00 ms |
DSK | sda | busy 101% | read 866 | write 197 | KiB/w 21 | MBr/s 1.38 | MBw/s 0.42 | avio 9.34 ms |
DSK | sdb | busy 98% | read 914 | write 407 | KiB/w 12 | MBr/s 1.53 | MBw/s 0.49 | avio 7.32 ms |

One of the main causes of this is courier (imap and pop3) and eventually statics collerctor, i am using qmail and files are .mbox, i am finding the way to reduce this disk IO problem.

 

[Cristian Rodriguez]

Today i wake up and again statics collector is causing high load in my server, maybe i should run every task separately to find which task is causing this issue.

One thing that call my atention is the fact that there is one task called spamtrain maybe it it is same as sa-learn, Should i delete sa-learn from cron.daily as it is a duplicate? spamassasin is going to be updated if i do this?

statics collector is causing all other tasks in cron to be delayed and being run at server high usage hours, so i want to run only the onces i need.

Do anyone know where can i find what every one of this process are doing? i can guess some of them, for example i do not need, Sitebuilder or some statics, or maybe i prefer sa-learn over ExecuteSpamtrain

CheckForUpdates
InstallUpdates
InstallSystemPackageUpdates
UpdateKeys
PleskUsage
Sitebuilder
MailUsage
ExecuteStatistics
ProcessAutoreports
OptimizeStatistics
AnalyseDomainStatistics
AnalyseClientStatistics
ExecuteSpamtrain
LoadCustomizations
UpdateApsCache
UpdateApsApplications
UpgradeExtensions
ExecuteGlCleaner
StoreProtectedConfigs
FileSharing
UpgradePanel
ExecuteWebStatistics
UpdateModSecurityRuleSet
AutoresponderEndDate
CheckPostponedFeedback
SearchIndex
ExecuteQuotacheck
UpdatePhpCurlCertificates
BackupRestoreStats
ComposerSelfUpdate

 

[Bitpalast]

Statistics should not cause a very high throughput unless the log files are very large. Maybe your log files are very large due to many errors or brute force attacks?

All maintenance tasks make sense. I do not think it would be good to deactivate maintenance tasks.

 

[Cristian Rodriguez]

It is very interesting, there is no information in internet about what every of the task listed with command below.

/usr/local/psa/bin/sw-engine-pleskrun /usr/local/psa/admin/plib/DailyMaintainance/script.php -l

I tried to find something but not explanation at all, i feel there is some kind of hermeticism with some topics related with Plesk, maybe nobody answer that question or sometimes the answer is just,

not i do not recomend you to do it

. maybe it is i what i feel, not only about this or other posts by me but any other i have read. I believe, if you are a customer yo have the right to know what your sistem is doing at least.

Half of the tasks in Daily Maintenance are not necesary for me, so i just want to omite them and others be executed by a cron job.

Related with the mater of this thread, maybe i have found part of the root of the problem, one website was using Wordpress W3 Cache and pages cache option was activated so it was creating a huge amount of files, i am deleting them with command rm -Rf cache/path and it is taking more than an hour and counting, i think is has deleted only the 15% or less of the existing files.

Is there a way to limit the amount of files allow per subscription in Plesk? i have seen that this option exist in WHM/CPanel. this is a good idea due to large amount of files affect server performance

 

[Cristian Rodriguez]

All this problem was caused by cache files created by Wordpress's plugin W3 Cache that created more than a million of files causing some Plesk and other process last too much, deleting all this files took at least 30 hours. Load average of server passed from almost 100 to 5 or less at high usage server hours.
I had some luck at identifying this issue due that i have more than 70 subscriptions in my server, I could identify this because i ran LMD (maldet) at that subscription particulary, i saw that the same process in other subscriptions took less than a minute and in this subscription it took a lot of time even when i excluded files older than three days.
Now the task are executed correctly and at the time indicated.
In this cases would be helpful to have a record of how many files exist in each subscription.