Memory usage really high.

[seanstephens]

I just did a top on my server and I found that the server was using the entire 2GB of memory and 1/2 the swap partition. However the CPU load average was under 1.0 . Aftert a reboot it returned to normal. This is not the first time that the server has slowed down to a crawl either. Anybody got any ideas what could be causing that? Or how I could find out what would be causing something like that?

Thanks for your time.

 

[CBiLL]

Are you sure those memory aren't "cache" memory? Linux caches a huge amount of rams that it look like it taking up all the ram but actually it's cache and only small percentage of memory within that cache is used up ..

CBiLL

 

[ttucker]

Mine Too

My server has 1 gig of memory and runs at about 98%. When I restart it, it goes back to normal (50% or so).

However when it reaches 99% or so, the pages that cause the most traffic return to the plesk default page.......and thats not good.

Any body have any ideas?

 

[bearkuma]

Solution?

Did you ever figure this out? I'm having the same problem except my swap space almost reaches 100% and the server is so slow it refuses to dish out any more pages and nobody can get their mail. I'd be very interested if you have a solution.

Thanks

Steve
Trend New Technologies

 

[etan]

what OS and version of plesk are you running, sounds like something similar that i have.

what processes use up all the swap ?

 

[bearkuma]

I'll check it out

I'm running Red Hat 9 with Plesk 7.0.2 build040430.16. I'm watching the processes right now, but everything seems fine (just rebooted 2 hours ago) It's a fine line to wait until things slow down to much because the phone starts ringing and clients start yelling. (that's when I panic and reboot ;-)

Here's how things look now. Server is still fast and all is good. I'll post again when I can see what's going on.

11:51:49 up 1:51, 4 users, load average: 0.24, 0.10, 0.16
147 processes: 144 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 2.5% user 0.5% system 0.0% nice 0.0% iowait 96.8% idle
Mem: 1030900k av, 660676k used, 370224k free, 0k shrd, 99436k buff
558672k actv, 16184k in_d, 5540k in_c
Swap: 5116660k av, 36496k used, 5080164k free 218024k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1316 root 15 0 61656 11M 2056 S 1.1 1.1 3:45 0 X
1584 root 15 0 13088 12M 10512 R 0.7 1.2 0:01 0 kdeinit
1570 root 15 0 15684 15M 8932 R 0.1 1.5 0:01 0 rhn-applet-gui
1575 root 15 0 14504 14M 11536 S 0.1 1.4 0:04 0 kdeinit
4531 root 15 0 1252 1252 868 R 0.1 0.1 0:08 0 top
1 root 15 0 104 96 56 S 0.0 0.0 0:04 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
.......

 

[sieb@]

Uninstall X/kde, it has no use on a production server, is a security risk, and is a resource hog.

Linux will allocate all of the available ram but hand it back when its needed by an app, its not actually "in use" by anything. Same as the "Idle process" in windows.

 

[bearkuma]

Things are still running fairly fast, but I can see the normal progression of things beginning already. Once the memory becomes fully used (or almost fully used) the swap space begins a gradual crawl up. Eventually, things become slower and slower to the point that web pages are no longer going out.

The used swap space is now up to 25% .... and climbing.

I'm quite new to all of this so any suggestions would be greatly appreciated!

Thanks,

Steve

09:53:20 up 23:52, 3 users, load average: 0.03, 0.10, 0.07
149 processes: 141 sleeping, 6 running, 2 zombie, 0 stopped
CPU states: 2.5% user 0.5% system 0.0% nice 0.0% iowait 96.8% idle
Mem: 1030900k av, 1013332k used, 17568k free, 0k shrd, 142780k buff
748012k actv, 0k in_d, 19912k in_c
Swap: 5116660k av, 193284k used, 4923376k free 571400k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1316 root 15 0 60336 5288 1144 R 2.1 0.5 4:54 0 X
1584 root 15 0 4544 3072 1968 R 0.7 0.2 0:06 0 kdeinit
4531 root 15 0 916 888 376 R 0.1 0.0 2:47 0 top
1 root 15 0 104 76 56 S 0.0 0.0 0:04 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0

 

[bearkuma]

Here it goes again!

As I stated in my earlier post, after a seemingly random amount of time (this time 10 days) the server is slowing way way down. I can see some odd looking information when I run top, but an unsure what I should do about it.

Any suggestions would be greatly appreciated! Thanks.

Physical Memory has 8MB free out of 1GB (App data 42%, Disk buffers 19%, Disk cache 37%)
Swap space is 51% with just less than 500MB to go.



16:33:37 up 10 days, 6:33, 3 users, load average: 6.53, 6.22, 5.99
329 processes: 300 sleeping, 11 running, 18 zombie, 0 stopped
CPU states: 84.2% user 15.7% system 0.0% nice 0.0% iowait 0.0% idle
Mem: 1030900k av, 994020k used, 36880k free, 0k shrd, 199276k buff
627956k actv, 111908k in_d, 14832k in_c
Swap: 5116660k av, 446332k used, 4670328k free 380180k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1008 popuser 15 0 21988 8592 616 S 99.9 0.8 1:19 0 spamd
11044 apache 25 0 2580 1056 708 R 23.8 0.1 71:14 0 perl
15069 apache 25 0 2116 432 240 R 22.6 0.0 2359m 0 perl
28137 apache 25 0 2344 888 468 R 22.4 0.0 3783m 0 perl
29238 apache 25 0 2716 2004 840 R 21.8 0.1 211:48 0 perl
1316 root 15 0 63488 6224 612 R 3.3 0.6 18:21 0 X
1584 root 15 0 4832 3332 2272 R 0.9 0.3 14:22 0 kdeinit
7 root 15 0 0 0 0 SW 0.5 0.0 12:29 0 kscand/Normal
22467 root 25 0 1124 1124 796 S 0.5 0.1 0:00 0 badmsg
4531 root 15 0 932 896 388 R 0.3 0.0 36:51 0 top
12142 apache 15 0 4784 4408 1752 S 0.3 0.4 0:03 0 perl
8 root 16 0 0 0 0 SW 0.1 0.0 2:02 0 kscand/HighMem
9086 apache 15 0 4892 2960 832 S 0.1 0.2 0:12 0 perl
12141 apache 15 0 5480 5064 1752 S 0.1 0.4 0:04 0 perl
1 root 15 0 104 76 56 S 0.0 0.0 0:04 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
5 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kswapd
6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/DMA
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
24 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
25 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
26 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
27 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
28 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
29 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
30 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 raid1d
31 root 25 0 0 0 0 SW 0.0

 

[sieb@]

Looks like its Spamassassin (spamd) bogarting CPU time, meaning its crunching on too much mail. I would install "qmhandle" and see how much mail is going through your queues. I would also check into what those Perl scripts are.. Could be spammers, or valid scripts getting taken advantage of. Thumb through the logs (/var/log and /usr/local/psa/var/log) and see if there's anything suspicious going on, like lots of smtp hits/rejects/a perl script being accessed alot/etc..

 

[bearkuma]

Spam Assassin

Thanks as always for the suggestions. I've been sifting through all the logs and haven't come across anything definitive (although I must admit I'm not 100% sure what I'm looking for)..... however, my question is - The processes that are listed in my previous post as user:apache and perl... is this spam assassin at work? I've tried to stop if from running to see if it makes a difference but it keeps coming back to life. I'm actually considering removing the rpms just to finally get it done. Any thoughts? Good idea, bad idea? If those apache/perl processes are really spam assassin I will probably give it a try.

Thanks,

Steve
Trend New Technologies

 

[sieb@]

Spamd is spamassassin, apache and perl aren't related unless its some sort of scripted mailer running on one of the domains. Otherwise I would assume apache/perl are normally high processes when the server is hosting alot of sites.

You can always try disabling Spamassassin from starting up (if your on a RH server): at the command prompt type: setup. Then go into services and disable spamassassin, then reboot.

 

[niel]

In GNU/Linux, you'll only run into trouble when your swap usage is reaching its limit.

I wouldn't pay much attention to ram usage as the kernel tries to use the maximum for performance.

 

[bearkuma]

Perl high CPU cause finally found

I finally got to the bottom of what was causing the multiple instances of perl to run continuously and bog down the cpu in my machine. This post is just to lay it all out for anyone who may also have encountered the same thing. I know before I even begin explaining that many experienced admins
eyes will roll when I explain the cause. I'm new at this, so be nice ;-)

Personally I found it funny that I could find only a few vague references to this problem scattered throughout this forum, am I the only one to have this problem I wonder.

Anyway, it appears that someone or something uploaded some perl script to my tmp directory and it was sent on it's merry way doing it's best to bother google.com. I found various references to this 'webwatchbot' I think it was called in several forums that pointed to an out of date phpBB containing a weakness that was being exploited. My original problem was on my old 7.0.2 server, but we deployed a brand new server running 7.1.7 about 2 weeks ago that fell victim to the same problem in less than a week (cue to roll eyes).

I have since done several things as outlined in the following thread to end this problem (doing ok so far - fingers crossed)

http://forum.sw-soft.com/showthread.php?s=&threadid=21159&highlight=mod+security
http://forum.sw-soft.com/showthread.php?s=&threadid=12697&highlight=perl+tmp


Steve - clueless about Linux but being forced to learn the hard way ;-)