Apache crashes

[pimpmedia]

Today when I created a new domain with Plesk 7.5.2 on freebsd 4.9 Apache Crashed.

error_log:
[Tue Apr 19 12:40:18 2005] [notice] SIGUSR1 received. Doing graceful restart
[Tue Apr 19 12:40:19 2005] [notice] Apache/1.3.33 (Unix) mod_python/2.7.10 Python/2.2.2 mod_webapp/1.2.0-dev mod_perl/1.29 mod_throttle/3.1.2 PHP/4.3.10 FrontPage/5.0.2.2510 mod_ssl/2.8.22 OpenSSL/0.9.7e configured -- resuming normal operations
[Tue Apr 19 12:40:19 2005] [notice] suEXEC mechanism enabled (wrapper: /usr/local/psa/apache/bin/suexec)
[Tue Apr 19 12:40:19 2005] [notice] Accept mutex: flock (Default: flock)
critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument
critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument
[Tue Apr 19 12:41:11 2005] [crit] (48)Address already in use: make_sock: could not bind to port 443
critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument
critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument

After killing the remaining HTTPD processed and started Apache again all works fine.

I know the creation of the domain does an apachectl graceful and then the errors comes...

This problem occurs one time in a week or two weeks, somthimes when there is a plesk backup running, the backup triggers also a gracefull.

And are these errors:
critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument
from Apache, or from Frontpage?

Does some one have a solution?

 

[jas8522]

Same Problem

I'm having the very same problem. This never happend before, then a client of mine started adding many domains. After about 8 of them, every domain he added caused apache to crash... with the very same critical_release() fail errors.

Did you ever figure this out? Or is there another thread that I couldn't find on this?
I'll post if I find anything to solve this!

I'm also on FreeBSD 4.9.. interesting

Jordan

 

[jas8522]

Possible fix

I found this info in the version 7 troubleshooting section:

I solved the problem.

I have to modify the Apache configuration /etc/httpd/conf/httpd.conf

Find:

MaxClients 300 ( or 150 ? )

Replace by:

MaxClients 30

And restart server .
/sbin/reboot

Later ca I have nobody more concerns with Apache for the moment.

I just set that up myself, and we'll see how it goes.

Now the problem with it is that if you've got more than 30 simultaneous connections (which would get used up quick with many browsers opening multiple connections these days), it may pose a problem. It will queue the remaining connections, but if enough get queued there may be connection delays.

Anyone able to confirm that? Or able to explain why more than 30 connections would cause a remote problem like that?

Jordan

 

[jas8522]

No go!

I set the maxclients var to 30 and the problem is still not resolved!

critical_acquire() failed: Invalid argument
critical_release() failed: Invalid argument

Those errors continue to come up before apache fails to respond. The ironic thing is that I have another FreeBSD Server running it a completely different location with Plesk, and it NEVER did this, until Today!

So both servers are doing it and creating some problematic downtime!

I'll keep looking, but if anyone else has a solution, please let me know!

Thanks,

Jordan

[Edit] It seems that these are functions of Mod_Throttle. This being the mod that controls the bandwidth limiting in Plesk by directory (or in this case by domain, as they are the directories). I guess the question remaining is what argument is being passed to mod_throttle from Plesk that ends up invalid, and why is this only happening on FreeBSD systems?

On the primary box, according to the log this (4 times now) has only happened when someone adds a domain or adjusts traffic limits, so making some assumptions out of this, I would say that when the traffic limits are set on the domains Plesk isn't passing the right variable to mod_throttle, or the variable isn't in the appropriate format.

This server has not been reset in many months, and from other posts I've read on this topic, it's a possibility that some resource is being used to it's max, thereby causing this error, but again it's just a guess.

 

[voodoochile]

Re: Possible fix

Originally posted by jas8522
I found this info in the version 7 troubleshooting section:

I solved the problem.

I have to modify the Apache configuration /etc/httpd/conf/httpd.conf

Find:

MaxClients 300 ( or 150 ? )

Replace by:

MaxClients 30

And restart server .
/sbin/reboot

Later ca I have nobody more concerns with Apache for the moment.

I just set that up myself, and we'll see how it goes.

Now the problem with it is that if you've got more than 30 simultaneous connections (which would get used up quick with many browsers opening multiple connections these days), it may pose a problem. It will queue the remaining connections, but if enough get queued there may be connection delays.

Anyone able to confirm that? Or able to explain why more than 30 connections would cause a remote problem like that?

Jordan

I only quoted this to say DO NOT DO THIS. Maxclients SHOULD NOT be set to 30.

This is a problem with mod_throttle, as that's what's generating the error messages.

It hinks that Apache is about to spin out off control and take the box down with it and consequently puts a choke on the webserver to keept hat from happening.

Now, I sent an email to PSA and I got the answer "Mod_Throttle is buggy, just remove the AddModule and LoadModule references to it in the httpd.conf file."

This is BS. I removed the mod_throttle instances and my box started getting choked and ended up taking an ugly DOS and dying.

There are one of two problems. 1. Since they staticly compiled in the threshholds to mod_throttle, someone set things WAY to low (if they did this, SHAME ON YOU SW-SOFT, BAD, BAD CODERS). You can actually tune mod_throttle through Apache, but I didnt' have any luck with their version of mod_throttle.
The other possibility is that something was modified with apache forcing it to use more resources, and trigger mod_throttle into throwing things into a critical aquire/release state.

As I've stated, I asked sw-soft about this (and the httpd.core files in my psa/rc.d/ directory) and they said it was a mod_throttle issue. Wrong, once mod_throttle was out of the loop, I was still getting core files from apache on restarts.

Anyway, this is a small window to type in an dI've forgotten what I wrote above, if it's a bit incoherent, I apologize. =)

edit : On FreeBSD I was getting error inv/ar/log/messages about max child per procs when I'd restart Apache. It tooke me about a week of kernel and apache tuning to sort this BS out. If you want, I can drop ya the changes I made to the kernel in order to allow for more memory/children/children per parent process in addition to allowing more semaphores for the system as well (I believe it's your box running out of semaphores and causing httpd to crash, once I upped this, I haven't had any problems.)

Lemme guess, if you do a 'ls -al /usr/local/psa/admin/logs/*mm* you get a bunch of stuff back, right? =)

Also, when you make apache cahnges, just do a /usr/local/psa/rc.d/httpd restart as opposed to restarting the whole server. This will also "fix" things temporarily if mod_throttle is acting up. Not a fix, but it has the same effect without pissing off all of your customers. =)

 

[jas8522]

Yeah, I quickly set maxclients back to 150 when it wasn't working at 30.

It's always wonderful to hear that a relatively nice product has people working on it that cannot accept responsibility for their coding problems, and thereby will not even attempt to fix them.

I haven't been restarting the box though as is - just killiing all apache processes and then starting them back up again. When I run /usr/local/psa/rc.d/httpd restart it dosen't come back up, so I have to kill 'em all to fix it! Also, this never seems to occur unless someone adjusts domain limits on any domain on the box (or adds a new domain - adjusting the new domain's limits). So not until mod_throttle get's called, then it goes nuts like you say.

I read elsewhere about FreeBSD boxes with Apache running out of semaphores - and yes the child processes error I do get, but I set a cron job up a while back to run '/usr/local/psa/rc.d/httpd restart ' every 24 hours, which would reset the number of child processes so that it wouldn't hit the max. I'm assuming that the child processes problem there is a slow acting problem that builds up towards max - which is why it takes a long time to create a real (crashing) problem. Whereas the mod_throttle critical_release() and acquire() problem is relatively fast - bout 5-10 mins of it and it goes down. So not only would 24 hours be too long to wait to restart the server in this case, but it won't work due to the processes needing to be killed.

I think you meant 'ls -al /usr/local/psa/apache/logs/*mm*' for the command, and yep, tons of files like that (in the apache logs, not httpsd logs).

Although I dont' have to restart the whole server, the fact that it takes me time before I notice that it's down, then to kill and reload apache, is annoying. The downtime slowly adds up that way, and customers get antsy ... for lack of a better word

Please do send your kernel changes this way, I can experiment with them on the other box for now anyway

Thanks for the help voodoochile,

Jordan

 

[jas8522]

Voodoochile?

Apache is continuing to act odd... today rather than disallowing connections, it decided to allow them, but point them to the normal httpdocs - as if there were no vhosts.

Wonderful.

Voodoochile if you've got those adjustments, let me know! Anyone else have anything to add?

Jordan

 

[voodoochile]

These are for a 4.9 series kernel, they'll be a bit different for a 5.3 series.
Also, set maxusers to 0 if you've got a decent chunk of ram. And a warning I have to give you is that you better have some decent hardware to back this. Where apache just craps out now, it's going to keep on trucking, and you better have the resources to let it. =)



#differing options-Non/standard/
makeoptions COPTFLAGS="-O2 -pipe -funroll-loops"
options ACCEPT_FILTER_DATA
options ACCEPT_FILTER_HTTP
options IPFIREWALL
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_VERBOSE
options PMAP_SHPGPERPROC=400
options NSWAPDEV=3
options QUOTA
options SEMMNI=256
options SEMMAP=768
options SEMMNU=768
options SEMOPM=2560
options SEMUME=256
options SEMMNS=393216
options SEMMSL=1536

edit: I'd also stronglyl suggest using the passive ipfw firewall that comes with FreeBSD as well. This will keep synfloods/scans impact down to a minimum.

 

[jas8522]

Still happening!

All of these variables were already set in a recompile done by the provider of the server about 6 months ago. So we reboot the server and kept watching it. A week passed, and no problems, but now yet again it is doing the very same thing. I add a domain and bang, apache dies - still listed in services list, but not actually allowing the viewing of any web sites.

Server is a P4 2.8 HT with 1GB RAM.

If anyone has any other solutions to this, please me know... seriously considering renting a Red Hat server and swapping everyone over to that to avoid these silly issues!

Jordan

 

[jas8522]

Solution

As many have said, and despite what others have thought to be the problem, it seems that mod_throttle is indeed the only issue.

I removed references to mod_throttle in apache's httpd.conf file and have not seen a single critic_release() or critical_acquire since.

If anyone knows what use mod_throttle has in Plesk, let me know - as I know that plesk's domain limits are still working without mod_throttle, so that wasn't the use of it. And apache dosen't seem to be going nuts for CPU usage, so it's not needed for keeping apache 'calmed down'.

Five days and no such problems!

Jordan

[EDIT]

Ok. Here's the deal. Mod_Throttle is a pussy. One sunny day came along and someone decided to get a huge spike in traffic to their site. Mod_throttle wasn't around to calm apache down, so apache went nuts, and the box went spinning out of control.

I re-enabled mod_throttle and lowered max clients in an attempt to keep the box up (not just apache, the entire box). Mod_Throttle started spitting out errors in the apache error_log yet again. Causing apache to die.

So it is an endless loop. I enable mod_throttle and it helps server load issues, but kills apache. I disable it and server load is higher than my buddies on cannabis day.

Time to switch to a windows server. May IIS save us from destruction.

[/Edit]

 

[Traged1]

This generally occurs when httpd or apache cannot close all child threads it has started.

To fix manually you can do the following:

fuser 443/tcp

That will list the child processes that are still alive.

Then do:

kill -9 xxxxxx xxxxx

Where xxxxxx is the thread number from the fuser output above.

To ensure that this occurs automatically you can edit your /etc/init.d/httpd file and change the following code block:

restart)
stop
start
;;

To:

restart)
stop
ps -ax | grep -v grep| grep httpd | grep -v restart | awk '{print "kill -9 " $1}' | sh
start
;;

 

[BSDdood]

IPFW config?

edit: I'd also stronglyl suggest using the passive ipfw firewall that comes with FreeBSD as well. This will keep synfloods/scans impact down to a minimum.[/QUOTE]

Hi Voodoo,

Can't seem to get Plesk to work with IPFW. Could you post your
rules please?