Facebook
Twitter
N
NetworkNinja
Guest
I've got a really weird problem. We have 3 servers all the same hardware, all are , CentOS 5.3, Plesk 8.6, Apache 2.2.3, 2GB Ram, 2GB Swap, Last 0, 5, 15 Proc: 0.65, 2.24, 2.60. Quite frequently these servers will start to lock up, plesk becomes extremely slow, and we have to reboot them. We cant seem to find any reason that these systems are locking up. we also tend to see a lot of dropped packets on the interface as well. Any ideas what we can look for or try to get this resolved?
We have tried to stop and start different services as the event is happening, it does not seem to fix the issue. Someone on another method of contact had mentioned that it was possible that an httpd process had gone rogue, which was also our first thoughts, via the control panel or service httpd stop; sleep 2; service httpd start; the problem continues until it gets so bad that the server(s) must be rebooted. We also considered that perhaps Spam Assassin, being the hog that it can be, was killing our systems, except that one of these 3 servers does not have Spam Assassin installed.
We have 3 different servers all of which are from the same mfg (ordered different times), same hardware, and same software configuration. Each of these 3 seem to have the same problem. Each of these systems have 100-150 clients (and about 120-200 domains). During the event, checking memory and processor load seems to show no change in the 'average' for our systems. We also noticed that ifconfig shows a HUGE (151Million or more) packets have been dropped since the last reboot [this was at 15 minutes after the last reboot, which was about an hour ago]. I believe that part of this is because of the firewall configuration dropping packets.
We have considered a few different options to diagnosing the problem, but are placing the problem out here because neither myself, nor my supervisor can figure out what could be causing this. Starting to think that this is one of two things. An stable, but vulnerable (or broken) kernel or, that this is a hardware issue.
I'll gladly give out any pertinent information requested to help you, help me.
Thanks for your time!
Kernel: Linux 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux
Release: CentOS release 5.3 (Final)
Memory output: [under 'normal' events]
total used free shared buffers cached
Mem: 2017 1494 523 0 136 690
-/+ buffers/cache: 667 1350
Swap: 2023 0 2023
Output of Top: [under 'normal' events]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3351 mysql 15 0 141m 26m 4752 S 0.3 1.3 0:50.62 mysqld
2759 root 17 0 69592 14m 10m S 0.0 0.7 0:01.77 mms
7228 apache 15 0 68584 41m 6416 S 0.0 2.0 0:03.43 httpd
4029 apache 16 0 66592 39m 6268 S 0.0 1.9 0:18.89 httpd
4033 apache 15 0 66548 38m 6056 S 0.0 1.9 0:18.98 httpd
4317 apache 15 0 66528 39m 6180 S 0.0 1.9 0:17.83 httpd
4028 apache 15 0 66500 39m 6296 S 0.3 1.9 0:20.69 httpd
4032 apache 15 0 66300 38m 6220 S 0.0 1.9 0:17.07 httpd
4520 apache 15 0 66264 38m 6256 S 0.0 1.9 0:18.80 httpd
4699 apache 15 0 66124 38m 6196 S 0.0 1.9 0:20.64 httpd
4772 apache 15 0 66096 37m 5488 S 0.0 1.9 0:13.92 httpd
4035 apache 15 0 65980 38m 6160 S 0.0 1.9 0:12.46 httpd
4140 apache 15 0 65960 38m 6148 S 0.0 1.9 0:16.68 httpd
4293 apache 15 0 65952 38m 6332 S 0.0 1.9 0:20.43 httpd
4034 apache 15 0 65888 38m 6260 S 0.0 1.9 0:15.99 httpd
4057 apache 15 0 65888 38m 5972 S 0.0 1.9 0:16.86 httpd
4987 apache 15 0 65868 38m 6240 S 0.0 1.9 0:22.73 httpd
4048 apache 15 0 65812 38m 6208 S 0.0 1.9 0:18.48 httpd
4030 apache 15 0 65784 38m 6224 S 0.0 1.9 0:15.36 httpd
7241 apache 15 0 65744 38m 5944 S 0.0 1.9 0:05.67 httpd
4058 apache 15 0 65708 38m 6316 S 0.0 1.9 0:18.31 httpd
4771 apache 15 0 65564 38m 6236 S 0.0 1.9 0:16.84 httpd
3075 named 20 0 50152 3624 1992 S 0.0 0.2 0:00.13 named
4163 psaadm 15 0 49668 21m 15m S 0.0 1.1 0:03.07 httpsd
3970 root 18 0 49292 25m 7024 S 0.0 1.3 0:00.58 httpd
4135 psaadm 15 0 49080 26m 20m S 0.0 1.3 0:03.03 httpsd
4119 root 18 0 43724 6704 4192 S 0.0 0.3 0:00.05 httpsd
13809 popuser 16 0 42872 37m 2308 S 8.3 1.9 0:46.05 spamd
19665 popuser 18 0 38876 33m 2304 S 0.0 1.7 0:18.49 spamd
3711 root 15 0 34320 29m 2408 S 0.0 1.4 0:04.97 spamd
4425 root 34 19 25616 9m 2132 S 0.0 0.5 0:00.07 yum-updatesd
3522 postgres 18 0 21084 3212 2788 S 0.0 0.2 0:00.76 postmaster
3633 postgres 15 0 21084 884 456 S 0.0 0.0 0:00.00 postmaster
4506 mailman 18 0 14032 8128 2880 S 0.0 0.4 0:00.76 python
2744 root 19 0 13464 3060 2556 S 0.0 0.1 0:00.02 acronisagent
4511 mailman 15 0 13336 7252 2796 S 0.0 0.4 0:00.38 python
2327 root 7 -8 13096 692 568 S 0.0 0.0 0:00.00 audispd
4504 mailman 20 0 13084 4764 620 S 0.0 0.2 0:00.00 mailmanctl
4509 mailman 18 0 13076 7284 2848 S 0.0 0.4 0:00.31 python
4512 mailman 18 0 13048 7044 2772 S 0.0 0.3 0:00.32 python
2517 dbus 15 0 12988 1084 844 S 0.0 0.1 0:00.03 dbus-daemon
4513 mailman 18 0 12984 6892 2740 S 0.0 0.3 0:00.24 python
4510 mailman 18 0 12968 6944 2744 S 0.0 0.3 0:00.29 python
4507 mailman 18 0 12948 6820 2712 S 0.0 0.3 0:00.26 python
4508 mailman 18 0 12924 6900 2744 S 0.0 0.3 0:00.28 python
2612 root 23 0 12856 1312 544 S 0.0 0.1 0:00.25 pcscd
2325 root 19 -4 12520 776 560 S 0.0 0.0 0:00.01 auditd
7869 root 15 0 12036 3168 2496 S 0.0 0.2 0:00.14 sshd
We have tried to stop and start different services as the event is happening, it does not seem to fix the issue. Someone on another method of contact had mentioned that it was possible that an httpd process had gone rogue, which was also our first thoughts, via the control panel or service httpd stop; sleep 2; service httpd start; the problem continues until it gets so bad that the server(s) must be rebooted. We also considered that perhaps Spam Assassin, being the hog that it can be, was killing our systems, except that one of these 3 servers does not have Spam Assassin installed.
We have 3 different servers all of which are from the same mfg (ordered different times), same hardware, and same software configuration. Each of these 3 seem to have the same problem. Each of these systems have 100-150 clients (and about 120-200 domains). During the event, checking memory and processor load seems to show no change in the 'average' for our systems. We also noticed that ifconfig shows a HUGE (151Million or more) packets have been dropped since the last reboot [this was at 15 minutes after the last reboot, which was about an hour ago]. I believe that part of this is because of the firewall configuration dropping packets.
We have considered a few different options to diagnosing the problem, but are placing the problem out here because neither myself, nor my supervisor can figure out what could be causing this. Starting to think that this is one of two things. An stable, but vulnerable (or broken) kernel or, that this is a hardware issue.
I'll gladly give out any pertinent information requested to help you, help me.
Thanks for your time!
Kernel: Linux 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux
Release: CentOS release 5.3 (Final)
Memory output: [under 'normal' events]
total used free shared buffers cached
Mem: 2017 1494 523 0 136 690
-/+ buffers/cache: 667 1350
Swap: 2023 0 2023
Output of Top: [under 'normal' events]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3351 mysql 15 0 141m 26m 4752 S 0.3 1.3 0:50.62 mysqld
2759 root 17 0 69592 14m 10m S 0.0 0.7 0:01.77 mms
7228 apache 15 0 68584 41m 6416 S 0.0 2.0 0:03.43 httpd
4029 apache 16 0 66592 39m 6268 S 0.0 1.9 0:18.89 httpd
4033 apache 15 0 66548 38m 6056 S 0.0 1.9 0:18.98 httpd
4317 apache 15 0 66528 39m 6180 S 0.0 1.9 0:17.83 httpd
4028 apache 15 0 66500 39m 6296 S 0.3 1.9 0:20.69 httpd
4032 apache 15 0 66300 38m 6220 S 0.0 1.9 0:17.07 httpd
4520 apache 15 0 66264 38m 6256 S 0.0 1.9 0:18.80 httpd
4699 apache 15 0 66124 38m 6196 S 0.0 1.9 0:20.64 httpd
4772 apache 15 0 66096 37m 5488 S 0.0 1.9 0:13.92 httpd
4035 apache 15 0 65980 38m 6160 S 0.0 1.9 0:12.46 httpd
4140 apache 15 0 65960 38m 6148 S 0.0 1.9 0:16.68 httpd
4293 apache 15 0 65952 38m 6332 S 0.0 1.9 0:20.43 httpd
4034 apache 15 0 65888 38m 6260 S 0.0 1.9 0:15.99 httpd
4057 apache 15 0 65888 38m 5972 S 0.0 1.9 0:16.86 httpd
4987 apache 15 0 65868 38m 6240 S 0.0 1.9 0:22.73 httpd
4048 apache 15 0 65812 38m 6208 S 0.0 1.9 0:18.48 httpd
4030 apache 15 0 65784 38m 6224 S 0.0 1.9 0:15.36 httpd
7241 apache 15 0 65744 38m 5944 S 0.0 1.9 0:05.67 httpd
4058 apache 15 0 65708 38m 6316 S 0.0 1.9 0:18.31 httpd
4771 apache 15 0 65564 38m 6236 S 0.0 1.9 0:16.84 httpd
3075 named 20 0 50152 3624 1992 S 0.0 0.2 0:00.13 named
4163 psaadm 15 0 49668 21m 15m S 0.0 1.1 0:03.07 httpsd
3970 root 18 0 49292 25m 7024 S 0.0 1.3 0:00.58 httpd
4135 psaadm 15 0 49080 26m 20m S 0.0 1.3 0:03.03 httpsd
4119 root 18 0 43724 6704 4192 S 0.0 0.3 0:00.05 httpsd
13809 popuser 16 0 42872 37m 2308 S 8.3 1.9 0:46.05 spamd
19665 popuser 18 0 38876 33m 2304 S 0.0 1.7 0:18.49 spamd
3711 root 15 0 34320 29m 2408 S 0.0 1.4 0:04.97 spamd
4425 root 34 19 25616 9m 2132 S 0.0 0.5 0:00.07 yum-updatesd
3522 postgres 18 0 21084 3212 2788 S 0.0 0.2 0:00.76 postmaster
3633 postgres 15 0 21084 884 456 S 0.0 0.0 0:00.00 postmaster
4506 mailman 18 0 14032 8128 2880 S 0.0 0.4 0:00.76 python
2744 root 19 0 13464 3060 2556 S 0.0 0.1 0:00.02 acronisagent
4511 mailman 15 0 13336 7252 2796 S 0.0 0.4 0:00.38 python
2327 root 7 -8 13096 692 568 S 0.0 0.0 0:00.00 audispd
4504 mailman 20 0 13084 4764 620 S 0.0 0.2 0:00.00 mailmanctl
4509 mailman 18 0 13076 7284 2848 S 0.0 0.4 0:00.31 python
4512 mailman 18 0 13048 7044 2772 S 0.0 0.3 0:00.32 python
2517 dbus 15 0 12988 1084 844 S 0.0 0.1 0:00.03 dbus-daemon
4513 mailman 18 0 12984 6892 2740 S 0.0 0.3 0:00.24 python
4510 mailman 18 0 12968 6944 2744 S 0.0 0.3 0:00.29 python
4507 mailman 18 0 12948 6820 2712 S 0.0 0.3 0:00.26 python
4508 mailman 18 0 12924 6900 2744 S 0.0 0.3 0:00.28 python
2612 root 23 0 12856 1312 544 S 0.0 0.1 0:00.25 pcscd
2325 root 19 -4 12520 776 560 S 0.0 0.0 0:00.01 auditd
7869 root 15 0 12036 3168 2496 S 0.0 0.2 0:00.14 sshd
