How To Filter Tagged Spam to a Spam IMAP Folder

[stupidnic]

Okay. It has taken me a little bit of time to figure out this process, but I have finally figured out how to do this.

First off, some disclaimers:

DO NOT ATTEMPT IF YOU DON'T KNOW WHAT YOU ARE DOING!
If you mess this up you could potentially stop delivery of messages to your users, so please understand what you are doing before you attempt this.

You must be using at least Plesk 8.1.1. This will NOT work for versions prior to this version.
I tested this all on Plesk 8.1.1, so I can only tell you that it works on that version. I know for a fact that it will NOT work on Plesk 8.0, because Plesk changed how the filtering occurs with spamc from version 8.0 to 8.1 (for the better thankfully).

Now that we have that stuff out of the way, here is how you can filter messages tagged as spam by spamassassin and move them to a folder other than the default POP3/main folder.

Things you are going to need:

Plesk's SpamAssassin module
maildrop

The way this process works right now is that Plesk makes changes to the users .qmail file to call psa-spamc with an argument to dictate if the message should be dropped (reject), or delivered (accept).

We are modifying the accept behavior to change how the messages are delivered. The reject method will still drop the message without delivering the message. The goal here is to work with Plesk rather than replacing what it does.

You are going to need to install maildrop on your server. The filtering script that I am going to provide below is written to use maildrop's filtering methodology.

I am running Debian so I just ran:

apt-get install maildrop

Another server I have is CentOS based and does not include maildrop in yum, so I had to compile it manually. If you have to do this, be sure to note the location of the maildrop binary, as you will need it later.

Once you have maildrop installed, you will need to install the following maildrop script.

import HOME
VHOME=$HOME

if ((/^X-Spam-Flag:.*YES/))
{
    `/usr/bin/test -d $VHOME/Maildir/.Spam/`
    if ( $RETURNCODE == 1 )
    {
        `/var/qmail/bin/maildirmake $VHOME/Maildir/.Spam/`
    }
    to "$VHOME/Maildir/.Spam/"
}

#If it isn't Spam or Virus, then deliver normally
to "$HOME/Maildir/"

You should put this file in /etc/mailfilter and make its owner popuser with group popuser (chown).

Now you need to modify the shell for the user popuser, otherwise the checks, and maildirmake commands will fail to execute properly and mail will not be delivered.

Change:

popuser:x:110:31:POP3 service user:/var/qmail/popuser:/sbin/nologin

This line can also end in /bin/false, you just want a valid shell for this user.

To:

popuser:x:110:31:POP3 service user:/var/qmail/popuser:/bin/bash

Last thing to change is the psa-spamc file itself.

We want to change the section:

if [ $check -ne 1 -o "$1" != "reject" ]; then
"$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
else
    retval=99
fi

To read:

if [ $check -ne 1 -o "$1" != "reject" ]; then
    FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
    if [ $FORWARD -ne 0 ]; then
        "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
    else
        /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
        retval=0
    fi
else
    retval=99
fi

IMPORTANT

Make a backup copy of this file. If you ever upgrade Plesk, it could be overwritten and you will need to make these changes again to get your spam filtering working once more.

Make sure that the path to maildrop is the correct path to maildrop and the same for the grep and wc binaries.

Also make sure that the path to 'test' is correct. The purpose of the 'test' line it so check for the existence of the .Spam folder and to create maildir if it doesn't.

Please let me know if you have any questions and I will attempt to answer them the best that I can.

Good luck.

 

[thewolf]

Hi,

Great post!

Will these changes automatically enable the quarantine folder feature for all mailboxes on the Plesk server?

Also, do you have a script that will send daily notifications to the users about their quarantine folder and clean up older messages after a certain amount of time?

If you could share it with us, that would be great.

Thanks.

 

[stupidnic]

These changes do not enable spam filtering for all accounts. It actually relies heavily on Plesk's SpamAssassin implementation to determine which users should have their mail filtered and which do not. The goal of the changes are to work with Plesk not work around it.

I have actually modified the scripts above to further tweak them as there was some functionality that I wanted to implement. Specifically discarding email over a predetermined threshold (ie: get rid of really spammy messages) as well as filtering for Viruses using ClamAV virus scanner.

Here is the latest versions of the scan script:

import HOME
import EXT
import HOST

if($SIZE < 2000000)
{
    xfilter "/usr/local/bin/clamscan.sh"
}

if ((/^X-Virus-Status:.*INFECTED/))
{
    `/usr/bin/test -d $HOME/Maildir/.Virus`
    if ( $RETURNCODE == 1 )
    {
        `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
    }
        to "$HOME/Maildir/.Virus/"
}

if ((/^X-Spam-Flag:.*YES/))
{
    if (/^X-Spam-Status: Yes, score=*!.* !.*/)
    {
        if ( $MATCH2 > 10 )
        {
                to "/dev/null"
        }
        else
        {
            `/usr/bin/test -d $HOME/Maildir/.Spam/`
            if ( $RETURNCODE == 1 )
            {
                `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
            }
            to "$HOME/Maildir/.Spam/"
        }
    }
}

#If it isn't Spam or Virus, then deliver normally
to "$HOME/Maildir/"

Here are the contents for the clamscan.sh file listed above:

#!/bin/bash
###############################################################################
# clamscan.sh by Tom Walsh                                                    #
# Created August 13th 2003                                                    #
# Modified August 27th 2003                                                   #
#                                                                             #
# This script was created to be used with maildrop. Maildrop expects any      #
# filter it calls to return the message back to it via stdout. clamscan does  #
# not do this, so the shell script acts as a wrapper for clamscan to make it  #
# follow this behavior.                                                       #
#                                                                             #
# We are also using ripmime to extract the attachments to be sure we catch    #
# any virus that might not be found in Maildir format.                        #
#                                                                             #
# The program locations are specific to freebsd, so if you aren't using       #
# freebsd, then you will need to modify the locations.                        #
#                                                                             #
# Dependancies:                                                               #
# Bash (duh)                                                                  #
# ClamAV (duh, again)                                                         #
# logger                                                                      #
# maildrop which include the reformail binary                                 #
###############################################################################


# Setup program locations

CLAM="/usr/bin/clamdscan"
REFORM="/usr/bin/reformail"
LOGGER="/usr/bin/logger"

TIME=$(/bin/date "+%s")
DIR="$TIME.$PPID"

MSG=$(/bin/cat /proc/self/fd/0)

SCAN=$(echo "$MSG" | $CLAM --stdout --disable-summary -)

# Capture the exit code to test for viruses

EXIT="$?"

# Was there a virus found?

if [ "$EXIT" == "1" ]; then

# If so... Lets grab what kind of virus it was
 VIRUS=$(echo "$SCAN" | grep "FOUND" | tail -n 1 | awk '{print $2}')

 # Now lets get the old Subject
 SUBJECT=$(echo "$MSG" | $REFORM -x Subject:)

 # Now lets alter it to indicate it is infected
 SUBJECT="**VIRUS** [$VIRUS] $SUBJECT"

 # Now add the altered Subject back to the original message
 MSG=$(echo "$MSG" | $REFORM -a"X-Virus-Status: INFECTED" -i"Subject: $(echo "$SUBJECT")")

 LOG=$($LOGGER -p mail.info -t clamav "$DIR Virus $VIRUS found and filtered.")

else

 # If there wasn't a virus, we still need to make note of that
 MSG=$(echo "$MSG" | $REFORM -a"X-Virus-Status: CLEAN")

 LOG=$($LOGGER -p mail.info -t clamav "$DIR Virus scan completed, no virus found.")

fi

# Send the altered message back to stdout
echo "$MSG"

# Alter exit code of the script to make maildrop behave as it should
exit 0

The code above requires clamAV be installed along with the reformail binary. You might need to edit the locations in the file to make it work with your setup.

 

[thewolf]

Hi,

Thanks for the updated script.

Do you happen to have a script that only does spam filtering (with a quarantine folder) and no virus filtering?

Thanks.

 

[stupidnic]

just remove this portion of the code from the mailfilter script above.

if($SIZE < 2000000)
{
    xfilter "/usr/local/bin/clamscan.sh"
}

if ((/^X-Virus-Status:.*INFECTED/))
{
    `/usr/bin/test -d $HOME/Maildir/.Virus`
    if ( $RETURNCODE == 1 )
    {
        `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
    }
        to "$HOME/Maildir/.Virus/"
}

That is what handles the Virus filtering. You can also just comment out the xfilter line with an #, and that will render the rest of the code that makes use of that xfilter line useless and not checked.

 

[thewolf]

Thanks, I'll give it a try asap.

Do you also have a script that will send daily notifications to the users with the list of the messages in their quarantine folder and clean up older messages from the quarantine after a certain amount of time has passed?

Thanks.

 

[stupidnic]

I don't have anything that would give you a list of messages in quarantine, but here is something I whipped up really quickly that would find messages older than 30 days in the .Spam folders and remove them.

It isn't a script, but more of a bash one liner that accomplishes the same task.

You can adapt it to work in a shell script rather easily as the logic is the same, just the script syntax is slightly different.

for i in `find /var/qmail/mailnames/ -type d -name ".Spam"`; do find $i -type f -mtime +30 | xargs rm -rf; done

 

[GARMTECH]

Great work, thanks!

 

[GARMTECH]

Do not forget to setup chmod go-rwx for mailfilter file.

Just one thought - will email quotas be enforced with such a filtering method?

 

[stupidnic]

Since the quotas are built into Plesk and this still works as a normal IMAP folder in Plesk... I would think that it does.

 

[Hal9000]

nice post!
i slightly changed the script as follows:

import HOME

if ((/^X-Spam-Flag:.*YES/))
{
`/usr/bin/test -d $HOME/Maildir/.Spam/`
if ( $RETURNCODE != 1 )
{
to "$HOME/Maildir/.Spam/"
}
}

to "$HOME/Maildir/"

Since there are users that use POP instead of IMAP, and I don't want the script to create a Spam folder if it doesn't exist, as for the POP users the messages will not appear.

 

[Hal9000]

actually, i have a problem
running plesk 8.3.0 on debian 4.0

i am getting this:

Dec 22 10:00:20 fonzie qmail-local-handlers[1529]: cannot reinject message to '[email protected]'
Dec 22 10:00:20 fonzie qmail: 1198314020.896369 delivery 1186: failure: /usr/bin/maildrop:_error_while_loading_shared_libraries:_libcourierauth.so.0:_cannot_open_shared_object_file:_No_such_file_or_directory
/

and this:

Dec 22 10:01:40 fonzie qmail-remote-handlers[1278]: starter: submitter[1279] exited on signal 15
Dec 22 10:01:40 fonzie qmail-remote-handlers[1278]: starter: submitter[1279] terminates abnormally with status 0
Dec 22 10:01:40 fonzie qmail-remote-handlers[1278]: cannot reinject message to mail system
Dec 22 10:01:40 fonzie qmail: 1198314100.644035 delivery 1183: deferral: qmail-remote_produced_no_output./
Dec 22 10:01:40 fonzie qmail: 1198314100.644108 status: local 0/10 remote 3/20 exitasap

any ideas for what the problem could be?

 

[Hal9000]

ooohh, i see... i failed to read this error while installing maildrop:

dpkg: error processing /var/cache/apt/archives/courier-authlib_0.58-4_i386.deb (--unpack):
trying to overwrite `/usr/share/man/man1/courierlogger.1.gz', which is also in package psa-courier-imap

now i removed that coureirlogger file but it still won't install

 

[Hal9000]

oh ****, i forced install but some core files got replaced haha

root@fonzie:~# dpkg -i --force-overwrite courier-authlib_0.58-4_i386.deb
(Reading database ... 237267 files and directories currently installed.)
Unpacking courier-authlib (from courier-authlib_0.58-4_i386.deb) ...
dpkg - warning, overriding problem because --force enabled:
trying to overwrite `/usr/share/man/man1/courierlogger.1.gz', which is also in package psa-courier-imap
dpkg - warning, overriding problem because --force enabled:
trying to overwrite `/usr/sbin/authenumerate', which is also in package psa-courier-imap
dpkg - warning, overriding problem because --force enabled:
trying to overwrite `/usr/sbin/courierlogger', which is also in package psa-courier-imap
Setting up courier-authlib (0.58-4) ...

as far as i can see, debian includes courier 4 but plesk courier 3... i am afraid of restarting courier now! and i cant download the psa package right now to reinstall it, cause packages have been removed from the autoupdate plesk server

 

[stupidnic]

Hal,

I am running maildrop under Debian 4.0 with no problems. I did upgrade up from Debian 3.1 and I am not running 8.3 though. Not sure if I can offer any assistance.

I do have to add that I currently have my mail filtering disabled because I encountered a problem after I upgraded to 8.2.1. For some reason mail forwards that were being filtered were not being forwarded on.

I am still looking into this issue, as it I am sure it has something to do with the return values.

 

[Hal9000]

i got the thing working in the meanwhile, in combination with horde ingo etc...
BUT i just discovered that mailgroups don't work properly with the modified psa-spamc file, the mails don't get delivered to the mail groups (if spam checking is enabled for them)! so something needs to be changed there, and hopefully it can be changed but i guess is that $QMAIL_LOCAL knows about mail groups, whereas maildrop doesnt

 

[stumyp]

How about adding this functionality to the PLESK distribution package ?
and additionally, it would be very helpful if spamassassin will learn from messages placed to Spam folder manually by user.
Thnaks

 

[stupidnic]

Progress Update

I have spent some time debugging the problems with mail groups (as mentioned by HAL) and mail forwards as I have noticed personally.

The problem is how we are delivering the files. Since maildrop has its own delivery mechanism, we are by passing the .qmail files for the user. This is where Plesk stores the mailgroups, and mail forwards for the users.... so when we by pass these .qmail files (since maildrop doesn't read them) we are by passing those additional delivery commands.

How do you fix it?

Well you have two options.

Option 1: Turn off spam filtering for any mailgroup or mail forwarding account.
Pros: Makes the delivery work as intended
Cons: No spam filtering

Option 2: Parse the .qmail files in maildrop and do additional logic based on the information in those files.
Pros: Will cause delivery to work as intended
Cons: More work for me to develop (code to parse the .qmail files properly)

For now I am going with option 1, as I have enough control over my users to turn off spam filtering for those users that managed to turn it on for a mail forward/mail group.

I am also actively working on getting something in place to handle the proper delivery options as defined in the .qmail file. I am sort of torn because I know there are qmail mail handler plugins that should be able to handle this sort of stuff in the Plesk API, I just can't seem to find the correct information in the API to tell me how to access it. The information is vauge at best, and down right obfuscated in other areas. <paranoia> It is almost like they don't want somebody to develop a plugin that would circumvent their SpamAssassin plugin, even though most of the functionality is there to do it.</paranoia>

Anyways... that is where we are for now. I will update more when I have something more to share.

 

[stumyp]

Thanks, stupidnic.
I've implented your solution with Plesk 8.3 and it works.

 

[stupidnic]

Quick Work Around for Mail Forwards and Mail Groups

Okay. I have a rather quick and dirty work around for the mail groups and mail forwards problem.

Here is the modifications that you need to make to the psa-spamc file:

On line 28 of the file add the following:

    FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
    if [ $FORWARD -ne 0 ]; then
        "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
    else
        /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
        retval=0
    fi

So the entire section should read:

if [ $check -ne 1 -o "$1" != "reject" ]; then
    FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
    if [ $FORWARD -ne 0 ]; then
        "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
    else
        /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
        retval=0
    fi
else
    retval=99
fi

Basically this is handing the delivery off to qmail-local if it detects any line in mailboxes .qmail file that begins with & (dot-qmail parlance for delivery to another mailbox). It fixes the problem with delivery to mail groups and mail forwards, and still filters the messages if they are to be delivered locally.

Just make sure that the paths to your grep and wc binaries are correct.

I could add some additional logic to it to detect more things, but at this point and time, it works well enough, so I am going to leave it at that. I will edit the script I posted on the first page to reflect these changes as well, just incase nobody gets to the second page of the thread.

Take care!