Thank you virtubox and trialotto
No cron or backup job is executed at that time.
I've a monitor process that use mod_status to bind a requested URI to the apache process, unluckly the 99% CPU apache process is not bind to any URI. Is there anything else i could use to monitor the apache process?
Find below my monitor/restart_if_high_load script.
I've checked the errors log, it has a lot of
Broken pipe: mod_fcgid: can't write spawn command
After the issue happened.
So following this article
After upgrade to Onyx error in website log: mod_fcgid: can't write spawn command
Can it be that?
I've update my FcgidMaxProcesses and other fcgid params (FcgidMaxProcesses previous was 30)
<IfModule mod_fcgid.c>
<IfModule !mod_fastcgi.c>
AddHandler fcgid-script fcg fcgi fpl
</IfModule>
FcgidIPCDir /var/run/mod_fcgid/sock
FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm
FcgidIdleTimeout 40
FcgidConnectTimeout 30
FcgidIOTimeout 45
FcgidInitialEnv RAILS_ENV production
FcgidIdleScanInterval 10
FcgidMaxProcesses 300
FcgidBusyTimeout 1800
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 50
FcgidProcessLifeTime 120
</IfModule>
FILE servermonit.sh
#!/bin/bash
# log
LASTRUN=/var/www/vhosts/example.com/servermonit/lastrun
LOGFILE=/var/www/vhosts/example.com/servermonit/load_kill_log
PSFILE=/var/www/vhosts/example.com/servermonit/ps_log
STATUSFILE=/var/www/vhosts/example.com/servermonit/status_log
STATUSFILE2=/var/www/vhosts/example.com/servermonit/status_log2
# obtain the server load
loadavg=`uptime |cut -d , -f 4|cut -d : -f 2`
thisloadavg=`echo $loadavg|awk -F \. '{print $1}'`
echo `date` - LOADAVG $loadavg - LOADAVGINT $thisloadavg > $LASTRUN
echo `date` - LOADAVG $loadavg - LOADAVGINT $thisloadavg
wget -O $STATUSFILE2
http://localhost/server-status
/var/www/vhosts/example.com/servermonit/httop -2 >> $STATUSFILE
if [ "$thisloadavg" -ge "1" ]; then
echo LOAD is high
ps auxfww >> $PSFILE
date >> $LOGFILE
/sbin/service httpd restart
fi
FILE httop
#!/usr/bin/perl
# The MIT License (MIT)
#
# Copyright (c) 2014 Razvan C. Cojocariu
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
##########################################
# CONFIG
##########################################
my $status_url = '
http://localhost/server-status'; # URL to the server-status
my $look_for = 'httpd'; # Name of the apache executable (so we can filter out the result from top)
my $top_cmd = 'top -b -n1'; # The 'top' command that gives us a full list of the processes
my $displ_cmd = 'watch -n 1 "'. $0 ." -1 | column -ts \Q$'\t'\E\"";
my $displ_cmd2 = $0 ." -1 | column -ts \Q$'\t'\E";
##########################################
# COMMAND LINE & IMPLEMENTATION
##########################################
if ($#ARGV >= 0 && !(grep $_ == $ARGV[0], ('-1', '-2')))
{
print "Syntax: httop [-1]\n";
exit;
}
elsif ($ARGV[0] == '-2') # If we need to simply execut the column command
{
exec $displ_cmd2;
exit;
}
elsif ($#ARGV == -1) # No arguments, so run the main command
{
exec $displ_cmd;
exit;
}
#######################
# Fetching the data
use LWP::Simple 'get';
# Get the top data
# The reason we're getting this first is that it takes more time than the http request!
my $top = `$top_cmd`;
# Quickly fetch the server status
my $status = get $status_url;
# If we failed, just exit
if (!defined $status)
{
die "Couldn't fetch the server status from '$status_url'";
}
###########################
# Now let's map the data
my %processes = ();
while ($top =~ /^\s*(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s*$/gmsi)
{
next unless ($12 eq $look_for);
$pid = $1;
$processes{$pid} = {
'PID' => $1,
'USER' => $2,
'CPU' => $9,
'MEM' => $10,
# To be set later
'IP' => '',
'DOMAIN' => '',
'REQUEST' => '',
};
{
my $val = $9;
$val =~ s/,/./;
$processes{$pid}{val} = $val;
}
}
################################################################
# Merge the info we got from apache with what we got from top
while ($status =~ /<tr><td><b>([^<]+)<\/b><\/td><td>(\d+)<\/td><td>([^<]+)<\/td><td>(?:<b>)?([^<]+)(?:<\/b>)?\s*<\/td><td>([^<]+)<\/td><td>([^<]+)<\/td><td>([^<]+)<\/td><td>([^<]+)<\/td><td>([^<]+)<\/td><td>([^<]+)\s*<\/td><td>([^<]+)<\/td><td nowrap>([^<]+)<\/td><td nowrap>([^<]+)<\/td><\/tr>/g)
{
my $pid = $2;
if (defined $processes{$pid})
{
$processes{$pid}{IP} = $11;
$processes{$pid}{DOMAIN} = $12;
$processes{$pid}{REQUEST} = $13;
$processes{$pid}{REQUEST} =~ s/\t/\s/g;
}
}
print "PID\tUSER\t\%CPU\t\%MEM\tIP\tDOMAIN\tREQUEST\n";
my @keys = sort { $processes{$b}{val} <=> $processes{$a}{val} } keys %processes;
my @items = @processes{@keys};
foreach $item (@items)
{
print $item->{PID}, "\t";
print $item->{USER}, "\t";
print pad($item->{CPU}, 4), "\t";
print pad($item->{MEM}, 4), "\t";
print $item->{IP}, "\t";
print $item->{DOMAIN}, "\t";
print $item->{REQUEST}, "\n";
}
sub pad
{
return (' ' x ($_[1] - (length $_[0]))) . $_[0];
}
Facebook
Twitter
