Urgent! /usr/local/psa/handlers/spool contains one huge 100GB file

[Timo002]

Hello,

I just got an email from the healthmonitor that my disk of the VPS with Pleks 12.5 is almost full.

So I was looking at it, and I saw that since ~12:45 something is eating my disk space.

Also the CPU went sky high at the same moment.

Looking at the processes, there was one php process that took a lot of recourses and was running for 95 minutes already. So I killed that process. It seems that it stopped eating my disk space.

After looking what are the large folders on my VPS, I found out that the folder `/usr/local/psa/handlers/spool` is about 100GB in size.

I found the folder by using this command:
# du -h / | grep '[0-9\.]\+G'

So, in the folder '/usr/local/psa/handlers/spool' there is only one file!

totaal 99137244
drwxrwx---. 2 popuser popuser         4096 mrt 22 15:54 .
drwxr-xr-x. 8 root    root            4096 aug 26  2015 ..
-rw-------. 1 popuser popuser 101516478670 mrt 22 15:46 messagezQPaLN

This one file is 100GB is size, and to free up some space, I have removed this file from my server.

What has happened?

At the moment the command sendmail.postfix is also very busy

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
21768 user  20   0 58476 2788 2056 R 74.0  0.1   6:30.55 sendmail.postfi

 

[PriyanA]

This could be related to 'hacked' (Password compromised) email address sending millions of emails with an attachment.

Check the postfix email log https://kb.plesk.com/en/111283

 

[Timo002]

@PriyanA, nothing special in maillog. Only a "heartbeat" that checks every minute if the SMTP server is still alive. And a couple of emails received / send.

 

[PriyanA]

found online, but it is qmail,

https://kb.plesk.com/en/122242

 

[trialotto]

@Timo002

In cases like yours, you should first have a look at the mail queue.

In general, the directory /usr/local/psa/handlers/spool/ is a temporary directory for mail services (Qmail or Postfix).

There are many reasons why this (temporary) directory fills up:

- a well-known issue with greylisting (patched some couple of years ago)
- hacked mail accounts (sending huge amounts of maill, that are rejected by the recipient mail server)
- hacked mail accounts (sending huge attachments, which cannot be processed by the local or external mail server)
- too large attachments (which are not processed, often due to some bad custom configuration)

and so on.

Naturally, cleaning the queue is the first step (since that also gives you additional insight into what has happened), followed by just removing data from /usr/local/psa/handlers/spool/.

In your situation, it seems to be the case that an attempt to send a "genuine mail" with too large attachments is present, with the attempt failing, resulting in a retry (over and over).

I can be mistaken, but I hope that I am not: this would simply mean that you can delete the mail in question from the queue.

You should be able to find details about the mail in the maillog, which information enables you to delete the mail in question from the queue.

Hope the above helps!

Regards

 

[PriyanA]

@trialotto, Nicely explained!

@Timi002 Panel > Tools & Setting > Mail Server Setting > Mail Queue

But Wonder how would want to send 100GB of attachments if not spaming

 

[trialotto]

@PriyanA

You stated

But Wonder how would want to send 100GB of attachments if not spaming

and, well, I have seen it happen: one mailbox sending a huge attachment, which didn´t work (rejected by the recipient mail server), resulting in the sending mail server retrying over and over again, while the owner of the mailbox thought "well, let´s try another 10 times" and, in the meantime, all other mail got stuck in the mail queue.

Sure, these cases are very rare, but they do happen, without any intention to spam.

A small tip: increase the maximum for attachments a little bit, or even better, use a dedicated mail server (a server with one objective: sending and receiving mail, with the additional and very considerable advantage that spam and/or viruses do not automatically enter the production environment, i.e. the server on which sites are hosted).

Regards.....

 

[Timo002]

Thnx for all help! But I don't think it is due to someone sending an email over and over again.

If you look at the screenshot of the disk usage, every 5 minutes the usage grows with approximately 2GB. And this repeats over and over, until I stopped a php process that was extreem high on resources.

Also the mailq was empty, checked it from plesk am checked it on CLI with 'mailq'.
The spool dir also had just one file, 100GB in size, and that file was probably growing an growing.

Maybe it's just a bad luck situation. I will do a deeper inspection on the maillog file at the moment the problems started, see if I can find something. But for now I don't see anything strange in it.

 

[PriyanA]

You have mention "until you stopped a php process" what was the process?

This could be related to sending mails via php sendmail.

Please check the Cron setup from all the users as well.

 

[trialotto]

@Timo002

It is rather logical that you will not find anything in the mail queue, that is essentially a different concept, in comparison to the temporary directory /usr/local/psa/handlers/spool.

Simply stated, the spool directory will be used if some mail related process is "hicking up".

You have stated already that some 2GB is added each time, which can be caused by a continuous loop in a mail related process, mails not being retrieved/removed or even that a specific user is on a high volume mail list or a mail list with some huge attachments. It can even be the case that a lot of internal forwards are causing this odd behaviour.

I am pretty sure that you should run the command find /var/qmail/mailnames -size +1G in order to find (read: get an indication of) the responsible mail account(s).

Note that the command above also applies to Postfix.

If you find a mailbox of that size (>1GB), then it is very likely that this mailbox is involved in/associated with the usage of disk space in the directory /usr/local/psa/handlers/spool.

You can then have a more detailed look in the maillog, by simply issue the command: grep -in5 <mailbox name in ""> /var/log/maillog

Note that this command shows the line number that includes the mailbox entry and, in addition, five additional lines (rather helpful).

Hope the above helps.

Regards.......