How do you monitor your servers?

[rspurlock]

Hi All,

I'm just curious for anyone that will share. We have several Plesk servers and it seems once in a blue moon, or at some certain time of day, the load goes crazy, we have 200+ processes and a huge number of mysql and httpd processes going. How does one go into a server and track down who the culprit is (if it is one or 2 sites)?

When I do a netstat, it just shows our domain name of the server itself and not the customer's domain most of the time. How do you tell which domain or which database is getting hammered? Is there software out there that will tell you? I'd rather move the culprit to a faster server than throw upgrades at it since the server have a fixed hard drive size.

Lastly, has anyone found a good guide to help you optimize the server for certain kinds of load (like heavy database usage)? Any help is appreciated and any monitoring suggestions are welcome. We have nagios ready to load up but haven't completed the setup so maybe we have a good start.

Thanks,

Rob

 

[lvalics]

Can be a lot of things to raise the load.

- Example: We installed mod_security and loaded rules and after this we saw a lot of load. Unfortunatelly in same time we upgraded to MySQl 4.1 and we thought that is a problem, all TOP commands showed MySQL.
After 2 weeks, we disallowed mod_security and all is back to normal.
- The ideea is that load can be a lot of things, from Database (use MyTop for MySQL) to Apache Dos Attacks (use APF firewall with BDF), try to look with tcpdump on eth what can cause problems and so on, is a hunting game sometimes.

 

[rspurlock]

Good idea

This particular server does run mod_security but I never tuned it or anything. I will try disabling it and see if the same problems exist. Anyone else have thoughts?

Thanks!

 

[sieb@]

I like to run ApacheTop and IFTop to watch what Apache is doing along with what kind of traffic is coming in over the eth interface (think netstat in real time).

Outside of that, you can use pstree or ps + its options to track down whats doing what. You can also do a "ls -lLA /proc/pid#" for more detail of a process id.

For the actual monitoring of my servers, I have snmp setup along with a dedicated server in our office running IPCheck to keep an eye on our hosting network. If anything fails or gets too high of a load, I get alerted.

Having APF w/ BFD keep an eye out for brut force attacks is nice, especially when it emails me telling me it blocked an offensive IP.

You could also use Tripwire to keep an eye on stuff. Though I haven't gotten around to implementing this myself.

 

[lvalics]

I will complete SIEB commentary a little bit.

http://rfxnetworks.net/sim.php - SIM is a system and services monitor for ‘SysVinit’ systems, working on Linux Systems.

IPCheck - http://www.paessler.com/ipcheck is for Windows Platform as I saw, correct me if I wrong.

http://host-tracker.com/ - you can subscribe to this services to get reports (SMS if you need) about your server uptime.

http://jeremy.zawodny.com/mysql/mytop/ - MySQL Top program

http://www.ex-parrot.com/~pdw/iftop/ - IFTop, display bandwidth usage on an interface.

http://www.webta.org/projects/apachetop/ - ApacheTOP

 

[rspurlock]

Good info

Thanks for the info guys. I'll get completing Nagios setup this weekend to keep an overall eye on things but the suggestions above should help a ton on a per server basis.

Not sure if I should start another thread on this but I have one particular server that is just freaking out. I will try removing mod_security and see if it helps and install SIM to try to restart services with alerts to help pinpoint the problem. Not really sure what it is. It seems to go crazy when using a java based WYSIWYG editor in Mambo/Joomla. When posting large articles, the server eventually freaks out. I can't get in remotely to see what's happening with the load but I'm sure it's load as I can still ping the server. It's running what I believe are the latest versions of ART's rpms for php, apache etc. Odd thing is it's not even live yet and only has 1 site so I'm pretty sure it's not an outside influence.

If anyone has run into an issue like this, any suggestions are welcome.

Thx,

Rob
LNHosting

 

[sieb@]

Yes, IPCheck is windows only, but I run it no a windows server in our office that does a few other things. Our actual hosting network is in a remote facility. I wanted a machine to watch the network remotely incase something happened. It does no good to have a monitoring server on the same network in most cases.

I tried Nagios but it was a pain to try and get working, where as IPCheck coupled with Passler's PRTG traffic monitoring software made for a quick and robust setup.