• If you are still using CentOS 7.9, it's time to convert to Alma 8 with the free centos2alma tool by Plesk or Plesk Migrator. Please let us know your experiences or concerns in this thread:
    CentOS2Alma discussion

Tomcat causes Apache to hang

L

linuxxphybrid

Guest
I have been having the problem of Apache's hanging for some time now. I made various changes to address the problem, but the problem has been reoccurring for the last 4 weeks or so, and I have to assume that nothing's really solved yet. After I looked at various set ups and configurations, I have come to suspect that Tomcat is causing the problem somehow. I see the following error messages before Apache hangs:

[Fri Dec 23 15:40:44 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:44 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:44 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:45 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:45 2005] [error] Cannot configure connection "ooen_net_Connection"
[Fri Dec 23 15:40:45 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:45 2005] [error] Cannot configure connection "ooen_net_Connection"
[Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection"
[Fri Dec 23 15:40:47 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:47 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:47 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:47 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:48 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:48 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:48 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:48 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:49 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:49 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
[Fri Dec 23 15:40:49 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:49 2005] [error] Cannot configure connection ooen_net_Connection
[Fri Dec 23 15:40:50 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:50 2005] [error] Cannot configure connection "ooen_net_Connection"
[Fri Dec 23 15:40:51 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:51 2005] [error] Cannot configure connection ooen_net_Connection
[Fri Dec 23 15:40:51 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Fri Dec 23 15:40:51 2005] [error] Cannot configure connection "ooen_net_Connection"
[Fri Dec 23 17:44:50 2005] [warn] child process 13865 still did not exit, sending a SIGTERM
[Fri Dec 23 17:44:50 2005] [warn] child process 13906 still did not exit, sending a SIGTERM
[Fri Dec 23 17:44:50 2005] [warn] child process 13935 still did not exit, sending a SIGTERM
[Fri Dec 23 17:44:50 2005] [warn] child process 13938 still did not exit, sending a SIGTERM

What can I really do to resolve the problem?
 
Apache hangs

I am also having apache hang and the only errors I see are also

[Sun Dec 25 11:40:43 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.4/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connect
ors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Sun Dec 25 11:40:43 2005] [error] Cannot configure connection "mydomain_com_au_Connection"

These errors occur some time before apache hangs so I don't know if they are related.

I am running
RHEL 4
Plesk 7.5.4
tomcat4
178 domains (35 using tomcat)

I am running
ps ax | grep -v grep | grep -c httpd
every 5 minutes. The number of httpd processes slowly rises and doesn't seem to go down unless there is a restart. I'm still studying this.

I want to see if I can restart apache at some point.

I have another server running
RHEL 3
Plesk 7.5.2
tomcat4
It doesn't have many domains but a number are using tomcat.

It doesn't have apache hangs yet.

Bill
 
Re: Apache hangs

Let's take a look into this together. This is too important.
Originally posted by billsutton
I am also having apache hang and the only errors I see are also

[Sun Dec 25 11:40:43 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.4/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connect
ors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
[Sun Dec 25 11:40:43 2005] [error] Cannot configure connection "mydomain_com_au_Connection"

These errors occur some time before apache hangs so I don't know if they are related.

Bill
Click to expand...
I checked these error messages and I also checked times when Apache hangs. The thing is that I do not always see Tomcat errors before Apache hangs. Two might be related, BUT it certainly does not mean that Tomcat error immediately triggers Apache to hang.

I looked for information about Apache's hanging, and I found the following:

http://forums.gentoo.org/viewtopic-t-151105-highlight-authdigestmodule.html
http://forums.gentoo.org/viewtopic-t-384660.html

Apparently, auth_digest_module has some problem. I am not so sure if this is related, but I commented out the module
Code: 
#LoadModule auth_digest_module modules/mod_auth_digest.so
to test.

They also discuss entropy (in OS), but I'm trying to find out what that means, since I don't know what it is.
 
This is the script I'm using:

#!/bin/bash
mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT
if [ $CNT -gt 99 ]; then
CNT=0
fi
if [ $CNT -le 9 ]; then
mn=`date`
echo $mn '- WARNING httpd needs to be restarted'
fi

I adapted it from http://forum.swsoft.com/showthread.php?s=&threadid=24957 .
I'm running it every 5 min from cron:
/home/admin/cron/checkapache.sh>>/home/admin/cron/log (Min=*/5 H,DM,M,DW=*)

Apache hung again early this morning. The relevant part of this script's log :
Wed Dec 28 02:00:01 EST 2005
22
21
21
20
16
15
10
8
8
8
8
8
Wed Dec 28 03:00:01 EST 2005
8

It was on 22 all night until 2am then dropped to 8 by 2:35. The last entries in apache logs was around 2:30

I'm going to try using the script to restart apache when the count drops to 8
#!/bin/bash
mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT
if [ $CNT -eq 8 ]; then
mn=`date`
echo $mn '- Apache is restarting'
/etc/rc.d/init.d/httpd restart
fi
 
Apache hangs one or two times per day so I guess I'll soon see if the script restarts it.

I did see the other thread. I haven't disabled auth_digest. See what happens for you. I think it relates to apache startups rather than hangs.

I'm on Plesk 7.5.4, RHEL4, httpd-2.0.52-12.2

Also have mod_auth_mysql-3.0.0 and 4PSA Advanced Statistics installed.

Have increased tomcat's memory in /etc/tomcat4/tomcat4.conf
CATALINA_OPTS="-Xms97m -Xmx128m -Djava.awt.headless=true"

Bill
 
Originally posted by billsutton
I did see the other thread. I haven't disabled auth_digest. See what happens for you. I think it relates to apache startups rather than hangs.

Bill
Click to expand...
Apache ran without any problem for about 22 hours, so I thought that this might have fixed the problem, but apparently this was not the case. It hanged again. At least I know that disabling auth_digest doesn't really fix this particular problem.
 
I haven't had apache hang over the last 24 hours. Had to reboot this morning because I was getting out of memory errors in tomcat, in particular with xslt processing.

See what happens today.
 
Just reviewed the thread again, and I found one thing that I don't quite get. Why are you restarting apache when the count (# of httpd processes) drops to 8? Is there any particular reason or you thought that it's an appropriate action following http://forum.swsoft.com/showthread.php?s=&threadid=24957 ? Also can we just monitor Apache(httpd)from shell script to see if Apache is responsive?
 
The last 2 times that apache had hung the number of httpd processes had dropped to 8.

I don't know how to monitor httpd responsiveness from a shell script. Also have to be careful not to interfere with normal shutdown and startup.

I wonder why not many are seeing this problem. Perhaps very few run plesk tomcat.
 
Originally posted by billsutton
I wonder why not many are seeing this problem. Perhaps very few run plesk tomcat.
Click to expand...
Slightly-off-topic:

My previous host was unable to get my JSP application up and running. I think that it was partly because they were incompetent and partly because most customers don't run Tomcat, so they just didn't have any experience. My current host isn't much better, but by the time I switched, at least I knew how to get it up and running.
 
I emailed swsoft and asked about this, since I am a licensed user. I don't know if I get any response though ...
 
My shell script restarted apache this afternoon. Here is part of the log the script produced:

Thu Dec 29 16:00:01 EST 2005
22
22
22
22
22
22
22
22
22
22
22
22
Thu Dec 29 17:00:01 EST 2005
22
20
11
9
9
8
Thu Dec 29 17:25:01 EST 2005 - Apache is restarting
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
16
16
16
16
16
17
Thu Dec 29 18:00:01 EST 2005
17

If you want to run this script I suggest you run the following from cron 1st to see what number of processes httpd drops down to:

#!/bin/bash
mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT
if [ $CNT -eq 8 ]; then
mn=`date`
echo $mn '- WARNING httpd needs to be restarted'
fi

Bill
 
Originally posted by billsutton
If you want to run this script I suggest you run the following from cron 1st to see what number of processes httpd drops down to:

#!/bin/bash
mn=`date +%M`
if [ $mn -eq 0 ]; then
mn=`date`
echo $mn
fi
CNT=`ps ax | grep -v grep | grep -c httpd`
echo $CNT
if [ $CNT -eq 8 ]; then
mn=`date`
echo $mn '- WARNING httpd needs to be restarted'
fi

Bill
Click to expand...
Would you help me to set this up? Do I set this up from Plesk -> Server -> Crontab or I do this differently?

Also (my) Apache has been up and running for longer than 24 hours straight now. It seems that disabling auth_digest has helped. This might be just a concidence, so I need to observer a little longer though.
 
I'm learning as I go too. This is what I did.

ssh into the server.
You are probably at /home/admin
mkdir cron
cd cron
vi checkapache.sh
type or paste the code. Watch out for the 'tick' characters that you use a tick and not a single quote.

chmod 755 checkapache.sh

Plesk -> server -> crontab -> add new task

Minute: */5
Hour: *
Day of month: *
Month: *
Day of week: *
Command: /home/admin/cron/checkapache.sh>>/home/admin/cron/log

It logs to /home/admin/cron/log

Just use the script that does not restart apache to find out first how many processes it drops to and if it is hanging at this point.

My script restarted apache again this morning.
 
I haven't had chance to set up the cron job yet, but Apache hanged again. I logged in, and ran the following command:
Code: 
bash
ps ax | grep -v grep | grep -c httpd
Then, the command returned 10. I restarted Apache and ran the following command again:
Code: 
ps ax | grep -v grep | grep -c httpd
The command returned 10. This means that I cannot just restart Apache just because # of httpd processes went down to 10.

What am I missing here? Am I understanding what's covered in this thread correctly?

Also how many IP addresses does your copy of Apache use?
 
I have 3 IPs.

I assume if I run the command immediately after restarting httpd I'd get 8 processes. However I host 180 domains and some are quite active. So after 5 minutes the number of processes is up to 12 or more.

If you run the logging only script for a while you might get a better picture of whats going on.

You could save the previous count somehow (I don't know how) and check if its greater than 10. That could stop continuous restarts.

Would be better to solve the real problem. What sort of java applications do you run? I have 2 main applications. One is an ecommerce app, nothing difficult, using mysql. The other is a content management system. It uses xslt which is a heavy memory user. The xslt is only run when changes are made but occassionally I get OutOfMemoryErrors. The tomcat memory config change suggested by plesk reduces the occurrence of this problem (see earlier in this thread).

I don't use jsp, only servets and most of the classes are stored in shared.
 
Originally posted by billsutton
Would be better to solve the real problem. What sort of java applications do you run? I have 2 main applications. One is an ecommerce app, nothing difficult, using mysql. The other is a content management system. It uses xslt which is a heavy memory user. The xslt is only run when changes are made but occassionally I get OutOfMemoryErrors. The tomcat memory config change suggested by plesk reduces the occurrence of this problem (see earlier in this thread).

I don't use jsp, only servets and most of the classes are stored in shared.
Click to expand...
Custom application. I wrote it. Uses JSP and pulls data from MySQL running on backend.
 
1. Having some trouble setting up a cron job, so I asked a question here; http://forum.swsoft.com/showthread.php?threadid=29885 .

2. I thought about this whole thing again, and if we set up a cron job to restart apache to address this problem (this is hopefully a temporary solution, of course), we should just check http (not the process but the response as we try to access http). I asked about it in some other forum, and the following should do it.
Code: 
echo "GET /" | nc localhost 80
I'm figuring out what the entire script should look like and how we should set up a cron job.

2.2. I also thought about # of httpd processes discussed in the following thread; http://forum.swsoft.com/showthread.php?threadid=24957 . I don't think that it makes much sense. MinSpareServers (in httpd.conf) defines # of minimum number of processes, and StartServers (in httpd.conf) defines # of processes at the start up. These two variables dictate # of processes ran, and I don't think that there is really any correlation between these numbers and Tomcat.
 
Update:

I wrote the following script to restart Apache upon it's hanging, locking up or shutting down.
Code: 
#!/bin/bash
LOGFILE="/home/admin/cron/http_log"

STATUS=`curl localhost --max-time 10`
if [ -z "$STATUS" ]; then
DATE=`date`
echo $DATE >> $LOGFILE
echo "Restart Apache" >> $LOGFILE
echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
fi
I think that something like this is what we should have for the time being. I actually want to modify the script and make it like the following.
Code: 
#!/bin/bash
LOGFILE="/home/admin/cron/http_log"

# Check server: Beginning
if [ (Check to see if uptime is longer than 10 minutes) ]; then

STATUS=`curl localhost --max-time 10`
# Check Apache: Beginning
if [ -z "$STATUS" ]; then
DATE=`date`
echo $DATE >> $LOGFILE
echo "Restart Apache" >> $LOGFILE
echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
# Email and notify
(Email and notify)
fi
# Check Apache: End
# Check memory: Beginning
if [(Check if memory used is over 1.5 GM; my server acts funny when memory used goes over 1.5GM)]
# Email and notify
(Email and notify)
(Wait until email gets sent)
# Reboot server
(Reboot server)
fi
# Check memory: End
fi
# Check server: End
With regard to debugging Tomcat, we may want to look into the following; http://forum.ev1servers.net/showthread.php?threadid=25376 .
 
You must log in or register to reply here.

Similar threads

L
Apache hangs ... (Tomcat is causing this?)
Replies
6
Views
2K
linuxxphybrid
L
Back
Top