1. Please take a little time for this simple survey! Thank you for participating!
    Dismiss Notice

Tomcat causes Apache to hang

Discussion in 'Plesk for Linux - 8.x and Older' started by linuxxphybrid, Dec 23, 2005.

  1. linuxxphybrid

    linuxxphybrid Guest

    0
     
    I have been having the problem of Apache's hanging for some time now. I made various changes to address the problem, but the problem has been reoccurring for the last 4 weeks or so, and I have to assume that nothing's really solved yet. After I looked at various set ups and configurations, I have come to suspect that Tomcat is causing the problem somehow. I see the following error messages before Apache hangs:

    [Fri Dec 23 15:40:44 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:44 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:44 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:45 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:45 2005] [error] Cannot configure connection "ooen_net_Connection"
    [Fri Dec 23 15:40:45 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:45 2005] [error] Cannot configure connection "ooen_net_Connection"
    [Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:46 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:46 2005] [error] Cannot configure connection "ooen_net_Connection"
    [Fri Dec 23 15:40:47 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:47 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:47 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:47 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:48 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:48 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:48 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:48 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:49 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:49 2005] [error] Cannot configure connection "ooen_net_Connection_ssl"
    [Fri Dec 23 15:40:49 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:49 2005] [error] Cannot configure connection ooen_net_Connection
    [Fri Dec 23 15:40:50 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:50 2005] [error] Cannot configure connection "ooen_net_Connection"
    [Fri Dec 23 15:40:51 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:51 2005] [error] Cannot configure connection ooen_net_Connection
    [Fri Dec 23 15:40:51 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.3/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connectors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Fri Dec 23 15:40:51 2005] [error] Cannot configure connection "ooen_net_Connection"
    [Fri Dec 23 17:44:50 2005] [warn] child process 13865 still did not exit, sending a SIGTERM
    [Fri Dec 23 17:44:50 2005] [warn] child process 13906 still did not exit, sending a SIGTERM
    [Fri Dec 23 17:44:50 2005] [warn] child process 13935 still did not exit, sending a SIGTERM
    [Fri Dec 23 17:44:50 2005] [warn] child process 13938 still did not exit, sending a SIGTERM

    What can I really do to resolve the problem?
     
  2. billsutton

    billsutton Guest

    0
     
    Apache hangs

    I am also having apache hang and the only errors I see are also

    [Sun Dec 25 11:40:43 2005] [error] Cannot read packet (//home/builder/pb_work_dir/psa_aiconfig_7.5.4/psa/apache-modules/mod_webapp/work/jakarta-tomcat-connect
    ors-4.1.29-src/webapp/lib/pr_warp_config.c:230)
    [Sun Dec 25 11:40:43 2005] [error] Cannot configure connection "mydomain_com_au_Connection"

    These errors occur some time before apache hangs so I don't know if they are related.

    I am running
    RHEL 4
    Plesk 7.5.4
    tomcat4
    178 domains (35 using tomcat)

    I am running
    ps ax | grep -v grep | grep -c httpd
    every 5 minutes. The number of httpd processes slowly rises and doesn't seem to go down unless there is a restart. I'm still studying this.

    I want to see if I can restart apache at some point.

    I have another server running
    RHEL 3
    Plesk 7.5.2
    tomcat4
    It doesn't have many domains but a number are using tomcat.

    It doesn't have apache hangs yet.

    Bill
     
  3. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Re: Apache hangs

    Let's take a look into this together. This is too important.
    I checked these error messages and I also checked times when Apache hangs. The thing is that I do not always see Tomcat errors before Apache hangs. Two might be related, BUT it certainly does not mean that Tomcat error immediately triggers Apache to hang.

    I looked for information about Apache's hanging, and I found the following:

    http://forums.gentoo.org/viewtopic-t-151105-highlight-authdigestmodule.html
    http://forums.gentoo.org/viewtopic-t-384660.html

    Apparently, auth_digest_module has some problem. I am not so sure if this is related, but I commented out the module
    Code:
    #LoadModule auth_digest_module modules/mod_auth_digest.so
    
    to test.

    They also discuss entropy (in OS), but I'm trying to find out what that means, since I don't know what it is.
     
  4. billsutton

    billsutton Guest

    0
     
    This is the script I'm using:

    #!/bin/bash
    mn=`date +%M`
    if [ $mn -eq 0 ]; then
    mn=`date`
    echo $mn
    fi
    CNT=`ps ax | grep -v grep | grep -c httpd`
    echo $CNT
    if [ $CNT -gt 99 ]; then
    CNT=0
    fi
    if [ $CNT -le 9 ]; then
    mn=`date`
    echo $mn '- WARNING httpd needs to be restarted'
    fi

    I adapted it from http://forum.swsoft.com/showthread.php?s=&threadid=24957 .
    I'm running it every 5 min from cron:
    /home/admin/cron/checkapache.sh>>/home/admin/cron/log (Min=*/5 H,DM,M,DW=*)

    Apache hung again early this morning. The relevant part of this script's log :
    Wed Dec 28 02:00:01 EST 2005
    22
    21
    21
    20
    16
    15
    10
    8
    8
    8
    8
    8
    Wed Dec 28 03:00:01 EST 2005
    8

    It was on 22 all night until 2am then dropped to 8 by 2:35. The last entries in apache logs was around 2:30

    I'm going to try using the script to restart apache when the count drops to 8
    #!/bin/bash
    mn=`date +%M`
    if [ $mn -eq 0 ]; then
    mn=`date`
    echo $mn
    fi
    CNT=`ps ax | grep -v grep | grep -c httpd`
    echo $CNT
    if [ $CNT -eq 8 ]; then
    mn=`date`
    echo $mn '- Apache is restarting'
    /etc/rc.d/init.d/httpd restart
    fi
     
  5. linuxxphybrid

    linuxxphybrid Guest

    0
     
  6. billsutton

    billsutton Guest

    0
     
    Apache hangs one or two times per day so I guess I'll soon see if the script restarts it.

    I did see the other thread. I haven't disabled auth_digest. See what happens for you. I think it relates to apache startups rather than hangs.

    I'm on Plesk 7.5.4, RHEL4, httpd-2.0.52-12.2

    Also have mod_auth_mysql-3.0.0 and 4PSA Advanced Statistics installed.

    Have increased tomcat's memory in /etc/tomcat4/tomcat4.conf
    CATALINA_OPTS="-Xms97m -Xmx128m -Djava.awt.headless=true"

    Bill
     
  7. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Apache ran without any problem for about 22 hours, so I thought that this might have fixed the problem, but apparently this was not the case. It hanged again. At least I know that disabling auth_digest doesn't really fix this particular problem.
     
  8. billsutton

    billsutton Guest

    0
     
    I haven't had apache hang over the last 24 hours. Had to reboot this morning because I was getting out of memory errors in tomcat, in particular with xslt processing.

    See what happens today.
     
  9. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Just reviewed the thread again, and I found one thing that I don't quite get. Why are you restarting apache when the count (# of httpd processes) drops to 8? Is there any particular reason or you thought that it's an appropriate action following http://forum.swsoft.com/showthread.php?s=&threadid=24957 ? Also can we just monitor Apache(httpd)from shell script to see if Apache is responsive?
     
  10. billsutton

    billsutton Guest

    0
     
    The last 2 times that apache had hung the number of httpd processes had dropped to 8.

    I don't know how to monitor httpd responsiveness from a shell script. Also have to be careful not to interfere with normal shutdown and startup.

    I wonder why not many are seeing this problem. Perhaps very few run plesk tomcat.
     
  11. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Slightly-off-topic:

    My previous host was unable to get my JSP application up and running. I think that it was partly because they were incompetent and partly because most customers don't run Tomcat, so they just didn't have any experience. My current host isn't much better, but by the time I switched, at least I knew how to get it up and running.
     
  12. linuxxphybrid

    linuxxphybrid Guest

    0
     
    I emailed swsoft and asked about this, since I am a licensed user. I don't know if I get any response though ...
     
  13. billsutton

    billsutton Guest

    0
     
    My shell script restarted apache this afternoon. Here is part of the log the script produced:

    Thu Dec 29 16:00:01 EST 2005
    22
    22
    22
    22
    22
    22
    22
    22
    22
    22
    22
    22
    Thu Dec 29 17:00:01 EST 2005
    22
    20
    11
    9
    9
    8
    Thu Dec 29 17:25:01 EST 2005 - Apache is restarting
    Stopping httpd: [ OK ]
    Starting httpd: [ OK ]
    16
    16
    16
    16
    16
    17
    Thu Dec 29 18:00:01 EST 2005
    17

    If you want to run this script I suggest you run the following from cron 1st to see what number of processes httpd drops down to:

    #!/bin/bash
    mn=`date +%M`
    if [ $mn -eq 0 ]; then
    mn=`date`
    echo $mn
    fi
    CNT=`ps ax | grep -v grep | grep -c httpd`
    echo $CNT
    if [ $CNT -eq 8 ]; then
    mn=`date`
    echo $mn '- WARNING httpd needs to be restarted'
    fi

    Bill
     
  14. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Would you help me to set this up? Do I set this up from Plesk -> Server -> Crontab or I do this differently?

    Also (my) Apache has been up and running for longer than 24 hours straight now. It seems that disabling auth_digest has helped. This might be just a concidence, so I need to observer a little longer though.
     
  15. billsutton

    billsutton Guest

    0
     
    I'm learning as I go too. This is what I did.

    ssh into the server.
    You are probably at /home/admin
    mkdir cron
    cd cron
    vi checkapache.sh
    type or paste the code. Watch out for the 'tick' characters that you use a tick and not a single quote.

    chmod 755 checkapache.sh

    Plesk -> server -> crontab -> add new task

    Minute: */5
    Hour: *
    Day of month: *
    Month: *
    Day of week: *
    Command: /home/admin/cron/checkapache.sh>>/home/admin/cron/log

    It logs to /home/admin/cron/log

    Just use the script that does not restart apache to find out first how many processes it drops to and if it is hanging at this point.

    My script restarted apache again this morning.
     
  16. linuxxphybrid

    linuxxphybrid Guest

    0
     
    I haven't had chance to set up the cron job yet, but Apache hanged again. I logged in, and ran the following command:
    Code:
    bash
    ps ax | grep -v grep | grep -c httpd
    
    Then, the command returned 10. I restarted Apache and ran the following command again:
    Code:
    ps ax | grep -v grep | grep -c httpd
    
    The command returned 10. This means that I cannot just restart Apache just because # of httpd processes went down to 10.

    What am I missing here? Am I understanding what's covered in this thread correctly?

    Also how many IP addresses does your copy of Apache use?
     
  17. billsutton

    billsutton Guest

    0
     
    I have 3 IPs.

    I assume if I run the command immediately after restarting httpd I'd get 8 processes. However I host 180 domains and some are quite active. So after 5 minutes the number of processes is up to 12 or more.

    If you run the logging only script for a while you might get a better picture of whats going on.

    You could save the previous count somehow (I don't know how) and check if its greater than 10. That could stop continuous restarts.

    Would be better to solve the real problem. What sort of java applications do you run? I have 2 main applications. One is an ecommerce app, nothing difficult, using mysql. The other is a content management system. It uses xslt which is a heavy memory user. The xslt is only run when changes are made but occassionally I get OutOfMemoryErrors. The tomcat memory config change suggested by plesk reduces the occurrence of this problem (see earlier in this thread).

    I don't use jsp, only servets and most of the classes are stored in shared.
     
  18. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Custom application. I wrote it. Uses JSP and pulls data from MySQL running on backend.
     
  19. linuxxphybrid

    linuxxphybrid Guest

    0
     
    1. Having some trouble setting up a cron job, so I asked a question here; http://forum.swsoft.com/showthread.php?threadid=29885 .

    2. I thought about this whole thing again, and if we set up a cron job to restart apache to address this problem (this is hopefully a temporary solution, of course), we should just check http (not the process but the response as we try to access http). I asked about it in some other forum, and the following should do it.
    Code:
    echo "GET /" | nc localhost 80
    
    I'm figuring out what the entire script should look like and how we should set up a cron job.

    2.2. I also thought about # of httpd processes discussed in the following thread; http://forum.swsoft.com/showthread.php?threadid=24957 . I don't think that it makes much sense. MinSpareServers (in httpd.conf) defines # of minimum number of processes, and StartServers (in httpd.conf) defines # of processes at the start up. These two variables dictate # of processes ran, and I don't think that there is really any correlation between these numbers and Tomcat.
     
  20. linuxxphybrid

    linuxxphybrid Guest

    0
     
    Update:

    I wrote the following script to restart Apache upon it's hanging, locking up or shutting down.
    Code:
    #!/bin/bash
    LOGFILE="/home/admin/cron/http_log"
    
    STATUS=`curl localhost --max-time 10`
    if [ -z "$STATUS" ]; then
    DATE=`date`
    echo $DATE >> $LOGFILE
    echo "Restart Apache" >> $LOGFILE
    echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
    fi
    
    I think that something like this is what we should have for the time being. I actually want to modify the script and make it like the following.
    Code:
    #!/bin/bash
    LOGFILE="/home/admin/cron/http_log"
    
    # Check server: Beginning
    if [ (Check to see if uptime is longer than 10 minutes) ]; then
    
    STATUS=`curl localhost --max-time 10`
    # Check Apache: Beginning
    if [ -z "$STATUS" ]; then
    DATE=`date`
    echo $DATE >> $LOGFILE
    echo "Restart Apache" >> $LOGFILE
    echo `/etc/rc.d/init.d/httpd restart` >> $LOGFILE
    # Email and notify
    (Email and notify)
    fi
    # Check Apache: End
    # Check memory: Beginning
    if [(Check if memory used is over 1.5 GM; my server acts funny when memory used goes over 1.5GM)]
    # Email and notify
    (Email and notify)
    (Wait until email gets sent)
    # Reboot server
    (Reboot server)
    fi
    # Check memory: End
    fi
    # Check server: End
    
    With regard to debugging Tomcat, we may want to look into the following; http://forum.ev1servers.net/showthread.php?threadid=25376 .
     
Loading...