Crashing system, memory error, and mod_python

[shunting]

After a restart of a dedicated virtual server running Plesk 8.3 on linux, here's an extract from /etc/httpd/logs/error_log 10 lines of either side of what I think is the trouble (***). Can anyone help me stabilize this?

[Thu Dec 25 10:22:15 2008] [error] [client 62.69.178.33] File does not exist: /var/www/vhosts/default/htdocs/agonist.rdf
[Thu Dec 25 10:22:34 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:22:38 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:22:45 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:22:58 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:22:59 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:23:00 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:23:15 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:23:16 2008] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Thu Dec 25 10:23:16 2008] [error] python_handler: no interpreter callback found.
[Thu Dec 25 10:23:20 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Thu Dec 25 10:23:21 2008] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Thu Dec 25 10:23:21 2008] [error] python_handler: no interpreter callback found.
[Thu Dec 25 10:23:21 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
*** [Thu Dec 25 10:23:22 2008] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Thu Dec 25 10:23:22 2008] [error] python_handler: no interpreter callback found.
zend_mm_heap corrupted
[Thu Dec 25 10:23:26 2008] [error] (12)Cannot allocate memory: fork: Unable to fork new process
Out of memory (Needed 130624 bytes)
[Thu Dec 25 10:23:27 2008] [error] [client 72.14.194.1] File does not exist: /var/www/vhosts/default/htdocs/favicon.ico
Out of memory (Needed 89804 bytes)
Out of memory (Needed 171444 bytes)
Out of memory (Needed 204100 bytes)
--

I don't think that the error is due to this "Directory index forbidden by Options" -- because that error has happened a lot before this crash (I just started looking at this host today).

Please, can anyone help?

 

[albanc]

I have the very same problem here :

[Tue Mar 24 10:55:01 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib64/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Tue Mar 24 10:55:01 2009] [error] python_handler: no interpreter callback found.
[Tue Mar 24 10:55:03 2009] [error] [client 193.167.100.161] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Tue Mar 24 10:56:07 2009] [error] [client 88.191.47.67] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Tue Mar 24 10:56:14 2009] [error] [client 87.106.38.250] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Tue Mar 24 10:56:43 2009] [error] [client 193.167.100.161] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind

It seems to occur when a system is scanned by script kiddies using the DFind vulnerability scanner. Notice all the occurences of

Directory index forbidden by Options directive: /var/www/html/

in your log. I have many of these too though they don't appear in this very section of the log, but many can be found not far in time in my log.

 

[albanc]

According to this thread, the issue occurs on Virtuozzo when the virtual host is out of memory. Xen is said not to be affected by this problem (could possibly parallels fix this ?)

I have checked my container setting and it had only 1 GB of memory allowed for hosting 29 websites, so it is possible that it is not enough. I am going to increase the allocated ressouces to see if things improve...

 

[albanc]

The server has been stable for 3 days for now (used to crash every 24 hours). I'm still continuing to cross fingers, but the issue seems to be solved now.

 

[psingh]

Crashing HTTPD only.. but everything is paralized

every 7 to 10 days following error occurs.. HTTPD service is down for about 5 to 7 minutes.. and wakes on it's own.

I am looking for permanent solution...

[Sun Nov 08 17:55:33 2009] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
Insufficient memory (case 4)
[Sun Nov 08 17:55:37 2009] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 08 17:55:42 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
more than 20 lines
[Sun Nov 08 18:01:38 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Nov 08 18:01:44 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Sun Nov 08 18:01:54 2009] [error] (12)Cannot allocate memory: fork: Unable to fork new process
FATAL: emalloc(): Unable to allocate 2152 bytes
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Sun Nov 08 18:07:15 2009] [error] python_handler: no interpreter callback found.
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Sun Nov 08 18:07:15 2009] [error] python_handler: no interpreter callback found.
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Sun Nov 08 18:07:15 2009] [error] python_handler: no interpreter callback found.
import time
ImportError: /usr/lib/python2.4/lib-dynload/timemodule.so: failed to map segment from shared object: Cannot allocate memory
[Sun Nov 08 18:07:15 2009] [error] python_handler: no interpreter callback found.
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
MemoryError
[Sun Nov 08 18:07:15 2009] [error] python_handler: no interpreter callback found.
[Sun Nov 08 18:07:15 2009] [error] make_obcallback: could not import mod_python.apache.\n
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 22, in ?




I am looking for permanent solution...


Can someone help...

 

[IgorG]

psingh,

Is it Virtuozzo container? Provide output of 'cat /proc/user_beancounters' in this case.
Did you checked RAM with memtest f.e. if it is standalone Linux server?

 

[albanc]

Hi psingh,

I guess you are using UBC memory management which is a piece of junk. Try switching the Memory Management Mode of your container to SLM ONLY (not SLM + UBC). This has solved my problem.

The problem with UBC is when it runs out of memory, instead of swapping it kills processes. SLM is much smarter and knows there are some processes which should not be wildly killed.

Regards,
Alban.

 

[IgorG]

Alban,

Yes, you are correct, but kernel OOM killer also can rage if SLMMEMORYLIMIT will be incorrectly calculated and configured for SLM based containers.

 

[psingh]

hi IgorG,

Thanks for your quick response.

This server is a VDS on Virtuozzo and following is response for :

# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
9644: kmemsize 10318245 10337554 16384000 18022400 47800491
lockedpages 0 0 650 650 0
privvmpages 197156 197997 262144 262144 759048
shmpages 9137 9137 131072 131072 0
dummy 0 0 0 0 0
numproc 115 115 425 425 0
physpages 41759 41769 0 2147483647 0
vmguarpages 0 0 65536 2147483647 0
oomguarpages 52171 52181 9223372036854775807 2147483647 0
numtcpsock 49 51 500 500 0
numflock 12 13 325 325 0
numpty 2 2 64 64 0
numsiginfo 0 1 512 512 0
tcpsndbuf 766840 815040 5365760 10485760 0
tcprcvbuf 660640 669424 5365760 10485760 0
othersockbuf 52944 86008 1503232 4063232 0
dgramrcvbuf 0 4392 235930 262144 0
numothersock 49 50 500 500 0
dcachesize 0 0 4194304 4317184 0
numfile 3806 3815 8192 8192 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 10 500 500 0



Thank you,
Sing

 

[IgorG]

psingh,

Yes there is really lack of important privvmpages resource. As you can see there is a big failcounters value.
Therefore I can suggest you increase it or switch to SLM memory management as it was already suggested previously.

 

[psingh]

thank you everyone for your valued effort.