Swap memory get full and crash the server.

[YoelM]

well this is the problem, My server is crashing eventually and I noticed that happens when the swap memory get full...

I have no clue what is causing this, and it started to happen after I upgrade to 9.5.1

Any idea how to monitor the swap memory and/or how to flush it if its necessary?

thanks

 

[IgorG]

First of all try to find with 'top' and 'ps ax' process which uses a maximum memory value. Investigate /var/log/messages and /var/log/sw-cp-server/error_log. Make sure that you haven't any hardware related problems on server.

 

[YoelM]

HI here is a random sample of my top

Tasks: 176 total, 7 running, 169 sleeping, 0 stopped, 0 zombie
Cpu(s): 43.1%us, 53.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 3.1%si, 0.0%st
Mem: 3691624k total, 2571864k used, 1119760k free, 179572k buffers
Swap: 4192880k total, 1516k used, 4191364k free, 732092k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2684 apache 15 0 431m 78m 5604 R 66.8 2.2 0:19.78 httpd
1107 apache 16 0 431m 83m 5000 R 42.4 2.3 6:34.62 httpd
1104 apache 15 0 426m 77m 5672 S 33.1 2.1 1:59.43 httpd
1171 apache 25 0 424m 76m 4548 R 22.8 2.1 1:56.89 httpd
1587 apache 25 0 425m 77m 5044 R 22.8 2.1 1:07.11 httpd
1095 root 15 0 12736 1152 808 R 3.5 0.0 1:34.07 top
1751 apache 25 0 420m 72m 4956 R 3.2 2.0 2:42.34 httpd
3125 root 15 0 38288 21m 3248 S 1.0 0.6 2:12.36 python
1106 apache 15 0 423m 75m 5032 S 0.6 2.1 3:06.97 httpd
1132 apache 16 0 422m 74m 5108 S 0.3 2.1 3:26.84 httpd
1165 apache 16 0 428m 80m 4996 S 0.3 2.2 4:05.78 httpd
1170 apache 15 0 426m 77m 5988 S 0.3 2.2 0:37.21 httpd
1368 apache 15 0 430m 80m 5836 S 0.3 2.2 3:29.29 httpd
1506 apache 15 0 426m 77m 5816 S 0.3 2.2 0:30.46 httpd
1508 apache 15 0 423m 74m 5724 S 0.3 2.1 0:34.83 httpd
1586 apache 15 0 426m 77m 5856 S 0.3 2.2 0:25.98 httpd
1 root 15 0 10344 684 572 S 0.0 0.0 0:00.81 init

in this case the swap it got a lot of room but suddenly it got full for somereason I dont know.
is the VIRT ok loading 423m of every httpd process?? it is MB

var/log/messages is showing nothng at all related with the swap memory.

an here is a copy of the /var/log/sw-cp-server/error_log

[root@ns2 ~]# cat /var/log/sw-cp-server/error_log
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
all children busy, launch additional (total 3, limit 30)
2010-05-03 15:18:19: (log.c.75) server started
2010-05-03 15:18:19: (mod_fastcgi.c.1337) Dynamic spawning with max_procs > 1 is not supported; setting min_procs = max_procs
2010-05-03 15:18:19: (mod_fastcgi.c.1337) Dynamic spawning with max_procs > 1 is not supported; setting min_procs = max_procs
2010-05-03 21:56:45: (mod_fastcgi.c.2582) FastCGI-stderr: PHP Warning: unlink(/usr/local/psa/tmp/components_dirty_flag) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /usr/local/psa/admin/auto_prepend/auth.php3 on line 504

2010-05-04 01:36:18: (log.c.75) server started
2010-05-04 01:36:18: (mod_fastcgi.c.1337) Dynamic spawning with max_procs > 1 is not supported; setting min_procs = max_procs
2010-05-04 01:36:18: (mod_fastcgi.c.1337) Dynamic spawning with max_procs > 1 is not supported; setting min_procs = max_procs

not sure if everything is because of the fastcgi module.

I dont use at all cgi on my sites, so probably would be good to disable it.

any sugegstions?

 

[IgorG]

Do you have any related errors in /var/log/httpd/error_log ?

 

[YoelM]

[error] (12)Cannot allocate memory: fork: Unable to fork new process

that's the last error just before the server crash.
Obviously is due to a full memory



In the mean time I have just setup an extra swap memory,
physical memory is 3.5GB
original swap partition 4Gb
and now I add 8GB swap memory file.(i got plenty room on my hard drive)

so total swap memory is 12GB right now.

I dont know how bad or good it could be but I will buy some time to get what could be the problem.


Also here is the ps aux | grep apache give me this

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 3471 0.0 1.1 282740 41664 ? S 14:10 0:00 /usr/sbin/httpd
apache 3501 2.1 12.7 838600 472404 ? S 14:10 3:05 /usr/sbin/httpd
apache 3502 1.1 2.6 460388 97440 ? S 14:10 1:39 /usr/sbin/httpd
apache 3503 2.2 13.0 844604 480240 ? S 14:10 3:11 /usr/sbin/httpd
apache 3505 3.2 11.8 842532 439168 ? S 14:10 4:35 /usr/sbin/httpd
apache 3506 1.1 4.3 525936 161368 ? S 14:10 1:39 /usr/sbin/httpd
apache 3665 0.9 2.1 444016 79180 ? S 14:10 1:24 /usr/sbin/httpd
apache 3849 2.2 13.1 850724 485056 ? S 14:10 3:08 /usr/sbin/httpd
apache 4038 1.1 2.6 462460 98212 ? S 14:15 1:37 /usr/sbin/httpd
apache 4039 1.1 2.1 441956 78664 ? S 14:15 1:29 /usr/sbin/httpd
apache 4041 1.2 2.1 440932 78480 ? S 14:15 1:38 /usr/sbin/httpd
apache 4042 1.0 2.1 444024 79272 ? S 14:15 1:26 /usr/sbin/httpd
apache 4043 1.0 2.1 444028 79320 ? S 14:15 1:23 /usr/sbin/httpd
apache 4378 1.2 1.9 433764 70488 ? S 14:28 1:34 /usr/sbin/httpd
apache 4381 2.4 4.3 522852 160000 ? S 14:28 3:03 /usr/sbin/httpd
apache 4384 2.3 2.1 446088 79328 ? S 14:28 2:50 /usr/sbin/httpd
apache 4387 1.2 4.3 525948 161048 ? S 14:28 1:29 /usr/sbin/httpd
apache 4390 1.0 2.6 460388 97468 ? S 14:28 1:20 /usr/sbin/httpd
apache 4391 1.0 2.1 441976 78060 ? S 14:28 1:17 /usr/sbin/httpd
apache 4867 2.5 13.0 845628 482136 ? S 14:44 2:45 /usr/sbin/httpd
apache 5513 1.1 2.1 444016 79184 ? S 15:04 1:00 /usr/sbin/httpd
apache 5814 1.1 2.1 444024 79804 ? S 15:17 0:50 /usr/sbin/httpd

rest of the process are not consuming too much memory or CPU, just the http.

 

[IgorG]

Do you have SMP kernel installed?

 

[atomicturtle]

apache 3501 2.1 12.7 838600 472404 ? S 14:10 3:05 /usr/sbin/httpd <- this is your culprit. Could be caused by a lot of things, like .htaccess files, or poor mod_rewrite rules