Tomcat causes Apache to hang

[billsutton]

My method of checking the number of httpd processes has been able to restart most failures however I have found the min number of processes to vary from 7 to 9 so far. So I will change over to using curl.

Has your script been able to restart httpd after a hang?

On server bootup crond is started after httpd so do you really need to check the uptime.

Bill

 

[linuxxphybrid]

Originally posted by billsutton
Has your script been able to restart httpd after a hang?

So far so good.

On server bootup crond is started after httpd so do you really need to check the uptime.

Bill

Definitely, ... so I need to modify the script slightly.

 

[billsutton]

In a Plesk FAQ they suggest adding

CATALINA_OPTS="-Xms97m -Xmx128m -Djava.awt.headless=true"

to /etc/tomcat4/tomcat4.conf.

A few days ago I increased Xmx from 128m to 256m. It went nearly 3 days without apache hanging and I don't think I had any OutOfMemory errors. Of course this could just be a change in client usage. Unfortunately it did eventually hang. The script detected it and restarted httpd but it didn't stop the hang this time. The script restarted apache every 5 min until I rebooted the server.

By the way, the shell script is now similar to yours. I'm still logging the number of httpd processes. It works well except in the above case.

LOGFILE="/home/admin/cron/log"
mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn >> $LOGFILE
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT >> $LOGFILE
STATUS=`curl localhost --max-time 10`
if [ -z "$STATUS" ]; then
mn=`date`
echo $mn "- Restart Apache" >> $LOGFILE
echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
fi

 

[linuxxphybrid]

Originally posted by billsutton
A few days ago I increased Xmx from 128m to 256m. It went nearly 3 days without apache hanging and I don't think I had any OutOfMemory errors. Of course this could just be a change in client usage. Unfortunately it did eventually hang. The script detected it and restarted httpd but it didn't stop the hang this time. The script restarted apache every 5 min until I rebooted the server.

I'm not 100% sure about this, but I seem to remember that Tomcat4, mod_webapps or both have a memory problem, and that's never been fixed. Tomcat focuses on developing 5.x and later versions and mod_webapps is basically out of maintenance. If this is the case, then allowing Tomcat to use a lot more memory would be certainly helpful. One thing that we should do is to detect the time when the server memory usage reaches a certain point. This is when we probably want to reboot the server in order to clean up memory preferably by running an automated script.

Say that we can keep the server up and running for 7 days straight. Say that Apache restarts once a day or so. If we can do both, then we can have at least 99% uptime. Is it great? Maybe, maybe not, but I think that this is satisfactory at least for the time being.

 

[Nikolai]

I think all these solutions are ridiculous. If you can't run tomcat without it hanging apache every day, then you should stop supporting it.

We have had the exact same problem, and we have decided to stop supporting it. We have told our existing customers to move hosts. 99.9% of our customers do not use tomcat, so we are not interesting in having an unreliable hosting services to satisfy 0.1% of our customers.

I would like to see Plesk drop mod_webapp and move to tomcat 5. Tomcat 4 is deprecated and the implementation into plesk is ****. Until then it is totally pointless offering jsp hosting.

 

[linuxxphybrid]

Originally posted by Nikolai
I think all these solutions are ridiculous. If you can't run tomcat without it hanging apache every day, then you should stop supporting it.

We have had the exact same problem, and we have decided to stop supporting it. We have told our existing customers to move hosts. 99.9% of our customers do not use tomcat, so we are not interesting in having an unreliable hosting services to satisfy 0.1% of our customers.

I would like to see Plesk drop mod_webapp and move to tomcat 5. Tomcat 4 is deprecated and the implementation into plesk is ****. Until then it is totally pointless offering jsp hosting.

As I mentioned in the thread, I run my own Java/JSP application, and this is why I needed to have some solution or workaround. I do agree that the solution is not elegant, nor is it perfect. It's up to each individual to decide if he/she should implement the solution or workaround presented here.

 

[wagnerch]

Originally posted by linuxxphybrid
I'm not 100% sure about this, but I seem to remember that Tomcat4, mod_webapps or both have a memory problem, and that's never been fixed.

Tomcat4 is actually very stable. I personally have never used mod_webapp, the favored connector is mod_jk. I have used mod_jk + tomcat4 and it is rock solid, lately been using mod_jk + tomcat5 without any issues.

 

[billsutton]

I am also running my own java application for hosting clients, using servlets only (no jsp). Its been migrated from JServ. It ran very well on JServ using much less memory. I suppose thats progress :-( but I wish hosting environments and shared applications were considered when the servlet containers were re-architected.

Anyway, since my change to increase the memory heap (5 posts ago), my application has been running smoothly - no outofmemory errors. However after 2 or 3 days apache hangs and a restart won't fix it. If I stop and start tomcat then restart apache then all is ok again. I suppose I'm running out of a resourse but don't know what yet.

I have altered the script to do that:

#!/bin/bash
LOGFILE="/home/admin/cron/log"

mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn >> $LOGFILE
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT >> $LOGFILE

STATUS=`curl localhost --max-time 10`
if [ -z "$STATUS" ]; then
mn=`date`
echo $mn "- Restart Tomcat and Apache" >> $LOGFILE
echo `/etc/rc.d/init.d/tomcat4 stop` >> $LOGFILE
sleep 10
echo `/etc/rc.d/init.d/tomcat4 start` >> $LOGFILE
sleep 5
echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
fi

This has worked twice now.

I would like to log other resources too. Any suggestions ? What commands ?

Bill

 

[linuxxphybrid]

Originally posted by billsutton
I would like to log other resources too. Any suggestions ? What commands ?

Bill

Should we check if restarting Tomcat is necessary and if it is not, just restart Apache?

I think that you and I both agree that we don't have to restart both Tomcat and Apache depending on a configuration. If so, isn't it more ideal if we just check to see if restarting Tomcat is necessary?

 

[billsutton]

After I increased tomcat's heap size I have not been able to get apache going by restarting it. It looked like it restarted ok but still couldn't access any websites. On the 1st occassion I just rebooted to get things going. On the 2nd occassion I tried restarting tomcat and then apache and that worked. I have not tried just restarting just tomcat.

Maybe its running out of some resource and restarting tomcat is freeing that resource. Doesn't seem to be running out of memory. Maybe descriptors ?

Bill

 

[linuxxphybrid]

Originally posted by billsutton
Maybe its running out of some resource and restarting tomcat is freeing that resource. Doesn't seem to be running out of memory. Maybe descriptors ?

Bill

descriptors???

 

[chiefdaddy]

Twice last week Apache stopped responding on my server... it just hangs.

If I restart Apache and Tomcat (or reboot the server) then it resumes working

It's weird that this just started happening as I've been running the server for about a year with no problems.

I'm running red hat enterprise with plesk 7.5 reloaded.

Is the problem with mod_webapp? Has anyone tried changing connectors?

 

[billsutton]

There is no other connector supported by this version of plesk. I haven't check the latest version. I still use a script (documented in this thread) to detect apache has hung and restart tomcat and apache.

Bill