Daily maintenance crashes server

[ehrenwert]

Hello,

I just mentioned that the daily maintenance crashed the server :-/ The load increased up to 144 (!) and RAM usage was 100%.

That's an excerpt of ps-output:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 29276 5568 ? Ss 00:06 0:02 /sbin/init
root 211 0.0 0.0 41152 16372 ? Ss 00:06 0:01 /lib/systemd/systemd-journald
root 481 0.0 0.0 25892 2356 ? Ss 00:06 0:00 /usr/sbin/cron -f
root 26366 0.0 0.0 56920 3964 ? S 01:53 0:00 \_ /usr/sbin/CRON -f
root 26369 0.0 0.0 4328 748 ? Ss 01:53 0:00 \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 26371 0.0 0.0 4216 656 ? S 01:53 0:00 \_ run-parts --report /etc/cron.daily
root 26372 0.0 0.0 4328 744 ? S 01:53 0:00 \_ /bin/sh /etc/cron.daily/50plesk-daily
root 26373 0.0 0.1 278356 54044 ? S 01:53 0:00 \_ /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/DailyMaintainance/script.php
root 29273 0.0 0.0 4328 740 ? S 01:53 0:00 \_ sh -c /opt/psa/bin/sw-engine-pleskrun "/opt/psa/admin/plib/DailyMaintainance/task-script.php" ExecuteStatistics '--period=daily'
root 29274 0.0 0.1 277072 52268 ? S 01:53 0:00 \_ /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/DailyMaintainance/task-script.php ExecuteStatistics --period=daily
root 29277 0.0 0.0 4328 752 ? S 01:53 0:00 \_ sh -c /opt/psa/admin/sbin/statistics --no-webstat
root 29278 0.0 0.1 278156 53400 ? S 01:53 0:00 \_ /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/sbin/statistics --no-webstat
root 29281 0.0 0.1 306072 76768 ? S 01:53 0:00 \_ /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/sbin/statistics --calculate-list --domain-ids 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200 no-webstat
root 29284 0.0 0.0 4328 868 ? S 01:53 0:00 \_ sh -c '/opt/psa/admin/bin/statistics_collector'
root 29285 0.0 0.0 60384 8552 ? D 01:53 0:00 \_ /opt/psa/admin/bin/statistics_collector

Any idea how to solve that?

 

[IgorG]

Is this VPS or standalone server?
Any related error messages in logs?

 

[ehrenwert]

It's running on VPS. But the box has 12 CPU cores and 40GB memory/120GB SWAP.

At the moment I'm running the daily scripts manually and see, that since the beginning of

root 22514 0.0 0.0 4328 768 pts/3 SN+ 10:59 0:00 sh -c /opt/psa/bin/sw-engine-pleskrun "/opt/psa/admin/plib/DailyMaintainance/task-script.php" UpdateApsApplications '--period=daily' '--process-wp-instance-ids=103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202'

root 22515 0.0 37.7 16050136 15820568 pts/3 RN+ 10:59 3:18 /usr/bin/sw-engine -c /opt/psa/admin/conf/php.ini /opt/psa/admin/plib/DailyMaintainance/task-script.php UpdateApsApplications --period=daily --process-wp-instance-ids=103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202

The memory-usage is growing very fast: http://d.pr/8u3C

 

[IgorG]

It's running on VPS. But the box has 12 CPU cores and 40GB memory/120GB SWAP.

Is it Virtuozzo VPS? If yes - check failcounters for different resources with

# cat /proc/user_beancounters

Many failcounters may cause kernel OOM Killer for VPS. This may be found on Virtuozzo node but not in VPS.

 

[ehrenwert]

It's proxmox and container runs on LXC.

I see that memory consumption is going up to 100% and the system begins swapping. After a while the memory-consumption goes back to normal level: http://d.pr/16mdW

 

[ehrenwert]

At the moment I see the following errors:

12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
12:24:38] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]

this happens in a loop.

Edit: Now the screen shows "KILLED" and load increases dramatically. The machine did not react to any signal and I had to reboot it.

I guess there is an error in wpmng.

If I try to sync the wordpress-installation on example.com via PLESK GUI, I'm getting this error:

[2015-12-18 14:26:08] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
[2015-12-18 14:26:08] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
[2015-12-18 14:26:08] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
[2015-12-18 14:26:08] ERR [util_exec] proc_close() failed ['/opt/psa/admin/bin/wpmng' '--user=example' '--php=/usr/bin/php5' '--' '--path=/var/www/vhosts/example.com/httpdocs' 'user' 'check-password' '' ''] with exit code [1]
[18-Dec-2015 14:26:08 Europe/Berlin] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /opt/psa/admin/externals/Zend/Db/Table/Row/Abstract.php on line 179