1. Please take a little time for this simple survey! Thank you for participating!
    Dismiss Notice
  2. Dear Pleskians, please read this carefully! New attachments and other rules Thank you!
    Dismiss Notice
  3. Dear Pleskians, I really hope that you will share your opinion in this Special topic for chatter about Plesk in the Clouds. Thank you!
    Dismiss Notice

How To Filter Tagged Spam to a Spam IMAP Folder

Discussion in 'Plesk for Linux - 8.x and Older' started by stupidnic, Aug 21, 2007.

  1. stupidnic

    stupidnic Guest

    0
     
    Okay. It has taken me a little bit of time to figure out this process, but I have finally figured out how to do this.

    First off, some disclaimers:

    DO NOT ATTEMPT IF YOU DON'T KNOW WHAT YOU ARE DOING!
    If you mess this up you could potentially stop delivery of messages to your users, so please understand what you are doing before you attempt this.

    You must be using at least Plesk 8.1.1. This will NOT work for versions prior to this version.
    I tested this all on Plesk 8.1.1, so I can only tell you that it works on that version. I know for a fact that it will NOT work on Plesk 8.0, because Plesk changed how the filtering occurs with spamc from version 8.0 to 8.1 (for the better thankfully).

    Now that we have that stuff out of the way, here is how you can filter messages tagged as spam by spamassassin and move them to a folder other than the default POP3/main folder.

    Things you are going to need:

    Plesk's SpamAssassin module
    maildrop

    The way this process works right now is that Plesk makes changes to the users .qmail file to call psa-spamc with an argument to dictate if the message should be dropped (reject), or delivered (accept).

    We are modifying the accept behavior to change how the messages are delivered. The reject method will still drop the message without delivering the message. The goal here is to work with Plesk rather than replacing what it does.

    You are going to need to install maildrop on your server. The filtering script that I am going to provide below is written to use maildrop's filtering methodology.

    I am running Debian so I just ran:

    apt-get install maildrop

    Another server I have is CentOS based and does not include maildrop in yum, so I had to compile it manually. If you have to do this, be sure to note the location of the maildrop binary, as you will need it later.

    Once you have maildrop installed, you will need to install the following maildrop script.

    Code:
    import HOME
    VHOME=$HOME
    
    if ((/^X-Spam-Flag:.*YES/))
    {
        `/usr/bin/test -d $VHOME/Maildir/.Spam/`
        if ( $RETURNCODE == 1 )
        {
            `/var/qmail/bin/maildirmake $VHOME/Maildir/.Spam/`
        }
        to "$VHOME/Maildir/.Spam/"
    }
    
    #If it isn't Spam or Virus, then deliver normally
    to "$HOME/Maildir/"
    
    You should put this file in /etc/mailfilter and make its owner popuser with group popuser (chown).

    Now you need to modify the shell for the user popuser, otherwise the checks, and maildirmake commands will fail to execute properly and mail will not be delivered.

    Change:
    Code:
    popuser:x:110:31:POP3 service user:/var/qmail/popuser:/sbin/nologin
    This line can also end in /bin/false, you just want a valid shell for this user.

    To:
    Code:
    popuser:x:110:31:POP3 service user:/var/qmail/popuser:/bin/bash
    Last thing to change is the psa-spamc file itself.

    We want to change the section:

    Code:
    if [ $check -ne 1 -o "$1" != "reject" ]; then
    "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
    else
        retval=99
    fi
    
    To read:

    Code:
    if [ $check -ne 1 -o "$1" != "reject" ]; then
        FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
        if [ $FORWARD -ne 0 ]; then
            "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
        else
            /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
            retval=0
        fi
    else
        retval=99
    fi
    
    IMPORTANT

    Make a backup copy of this file. If you ever upgrade Plesk, it could be overwritten and you will need to make these changes again to get your spam filtering working once more.

    Make sure that the path to maildrop is the correct path to maildrop and the same for the grep and wc binaries.

    Also make sure that the path to 'test' is correct. The purpose of the 'test' line it so check for the existence of the .Spam folder and to create maildir if it doesn't.

    Please let me know if you have any questions and I will attempt to answer them the best that I can.

    Good luck.
     
  2. thewolf

    thewolf Regular Pleskian

    25
    57%
    Joined:
    Mar 11, 2004
    Messages:
    231
    Likes Received:
    0
    Hi,

    Great post!

    Will these changes automatically enable the quarantine folder feature for all mailboxes on the Plesk server?

    Also, do you have a script that will send daily notifications to the users about their quarantine folder and clean up older messages after a certain amount of time?

    If you could share it with us, that would be great.

    Thanks.
     
  3. stupidnic

    stupidnic Guest

    0
     
    These changes do not enable spam filtering for all accounts. It actually relies heavily on Plesk's SpamAssassin implementation to determine which users should have their mail filtered and which do not. The goal of the changes are to work with Plesk not work around it.

    I have actually modified the scripts above to further tweak them as there was some functionality that I wanted to implement. Specifically discarding email over a predetermined threshold (ie: get rid of really spammy messages) as well as filtering for Viruses using ClamAV virus scanner.

    Here is the latest versions of the scan script:

    Code:
    import HOME
    import EXT
    import HOST
    
    if($SIZE < 2000000)
    {
        xfilter "/usr/local/bin/clamscan.sh"
    }
    
    if ((/^X-Virus-Status:.*INFECTED/))
    {
        `/usr/bin/test -d $HOME/Maildir/.Virus`
        if ( $RETURNCODE == 1 )
        {
            `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
        }
            to "$HOME/Maildir/.Virus/"
    }
    
    if ((/^X-Spam-Flag:.*YES/))
    {
        if (/^X-Spam-Status: Yes, score=*!.* !.*/)
        {
            if ( $MATCH2 > 10 )
            {
                    to "/dev/null"
            }
            else
            {
                `/usr/bin/test -d $HOME/Maildir/.Spam/`
                if ( $RETURNCODE == 1 )
                {
                    `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
                }
                to "$HOME/Maildir/.Spam/"
            }
        }
    }
    
    #If it isn't Spam or Virus, then deliver normally
    to "$HOME/Maildir/"
    
    Here are the contents for the clamscan.sh file listed above:

    Code:
    #!/bin/bash
    ###############################################################################
    # clamscan.sh by Tom Walsh                                                    #
    # Created August 13th 2003                                                    #
    # Modified August 27th 2003                                                   #
    #                                                                             #
    # This script was created to be used with maildrop. Maildrop expects any      #
    # filter it calls to return the message back to it via stdout. clamscan does  #
    # not do this, so the shell script acts as a wrapper for clamscan to make it  #
    # follow this behavior.                                                       #
    #                                                                             #
    # We are also using ripmime to extract the attachments to be sure we catch    #
    # any virus that might not be found in Maildir format.                        #
    #                                                                             #
    # The program locations are specific to freebsd, so if you aren't using       #
    # freebsd, then you will need to modify the locations.                        #
    #                                                                             #
    # Dependancies:                                                               #
    # Bash (duh)                                                                  #
    # ClamAV (duh, again)                                                         #
    # logger                                                                      #
    # maildrop which include the reformail binary                                 #
    ###############################################################################
    
    
    # Setup program locations
    
    CLAM="/usr/bin/clamdscan"
    REFORM="/usr/bin/reformail"
    LOGGER="/usr/bin/logger"
    
    TIME=$(/bin/date "+%s")
    DIR="$TIME.$PPID"
    
    MSG=$(/bin/cat /proc/self/fd/0)
    
    SCAN=$(echo "$MSG" | $CLAM --stdout --disable-summary -)
    
    # Capture the exit code to test for viruses
    
    EXIT="$?"
    
    # Was there a virus found?
    
    if [ "$EXIT" == "1" ]; then
    
    # If so... Lets grab what kind of virus it was
     VIRUS=$(echo "$SCAN" | grep "FOUND" | tail -n 1 | awk '{print $2}')
    
     # Now lets get the old Subject
     SUBJECT=$(echo "$MSG" | $REFORM -x Subject:)
    
     # Now lets alter it to indicate it is infected
     SUBJECT="**VIRUS** [$VIRUS] $SUBJECT"
    
     # Now add the altered Subject back to the original message
     MSG=$(echo "$MSG" | $REFORM -a"X-Virus-Status: INFECTED" -i"Subject: $(echo "$SUBJECT")")
    
     LOG=$($LOGGER -p mail.info -t clamav "$DIR Virus $VIRUS found and filtered.")
    
    else
    
     # If there wasn't a virus, we still need to make note of that
     MSG=$(echo "$MSG" | $REFORM -a"X-Virus-Status: CLEAN")
    
     LOG=$($LOGGER -p mail.info -t clamav "$DIR Virus scan completed, no virus found.")
    
    fi
    
    # Send the altered message back to stdout
    echo "$MSG"
    
    # Alter exit code of the script to make maildrop behave as it should
    exit 0
    
    The code above requires clamAV be installed along with the reformail binary. You might need to edit the locations in the file to make it work with your setup.
     
  4. thewolf

    thewolf Regular Pleskian

    25
    57%
    Joined:
    Mar 11, 2004
    Messages:
    231
    Likes Received:
    0
    Hi,

    Thanks for the updated script.

    Do you happen to have a script that only does spam filtering (with a quarantine folder) and no virus filtering?

    Thanks.
     
  5. stupidnic

    stupidnic Guest

    0
     
    just remove this portion of the code from the mailfilter script above.

    Code:
    if($SIZE < 2000000)
    {
        xfilter "/usr/local/bin/clamscan.sh"
    }
    
    if ((/^X-Virus-Status:.*INFECTED/))
    {
        `/usr/bin/test -d $HOME/Maildir/.Virus`
        if ( $RETURNCODE == 1 )
        {
            `/var/qmail/bin/maildirmake $HOME/Maildir/.Spam/`
        }
            to "$HOME/Maildir/.Virus/"
    }
    
    That is what handles the Virus filtering. You can also just comment out the xfilter line with an #, and that will render the rest of the code that makes use of that xfilter line useless and not checked.
     
  6. thewolf

    thewolf Regular Pleskian

    25
    57%
    Joined:
    Mar 11, 2004
    Messages:
    231
    Likes Received:
    0
    Thanks, I'll give it a try asap.

    Do you also have a script that will send daily notifications to the users with the list of the messages in their quarantine folder and clean up older messages from the quarantine after a certain amount of time has passed?

    Thanks.
     
  7. stupidnic

    stupidnic Guest

    0
     
    I don't have anything that would give you a list of messages in quarantine, but here is something I whipped up really quickly that would find messages older than 30 days in the .Spam folders and remove them.

    It isn't a script, but more of a bash one liner that accomplishes the same task.

    You can adapt it to work in a shell script rather easily as the logic is the same, just the script syntax is slightly different.

    Code:
    for i in `find /var/qmail/mailnames/ -type d -name ".Spam"`; do find $i -type f -mtime +30 | xargs rm -rf; done
    
     
  8. GARMTECH

    GARMTECH Guest

    0
     
    Great work, thanks!
     
  9. GARMTECH

    GARMTECH Guest

    0
     
    Do not forget to setup chmod go-rwx for mailfilter file.

    Just one thought - will email quotas be enforced with such a filtering method?
     
  10. stupidnic

    stupidnic Guest

    0
     
    Since the quotas are built into Plesk and this still works as a normal IMAP folder in Plesk... I would think that it does.
     
  11. Hal9000

    Hal9000 Guest

    0
     
    nice post!
    i slightly changed the script as follows:

    Since there are users that use POP instead of IMAP, and I don't want the script to create a Spam folder if it doesn't exist, as for the POP users the messages will not appear.
     
  12. Hal9000

    Hal9000 Guest

    0
     
    actually, i have a problem :(
    running plesk 8.3.0 on debian 4.0

    i am getting this:
    and this:

    any ideas for what the problem could be? :(
     
  13. Hal9000

    Hal9000 Guest

    0
     
    ooohh, i see... i failed to read this error while installing maildrop:

    now i removed that coureirlogger file but it still won't install :(
     
  14. Hal9000

    Hal9000 Guest

    0
     
    oh ****, i forced install but some core files got replaced haha
    as far as i can see, debian includes courier 4 but plesk courier 3... i am afraid of restarting courier now! and i cant download the psa package right now to reinstall it, cause packages have been removed from the autoupdate plesk server :(
     
  15. stupidnic

    stupidnic Guest

    0
     
    Hal,

    I am running maildrop under Debian 4.0 with no problems. I did upgrade up from Debian 3.1 and I am not running 8.3 though. Not sure if I can offer any assistance.

    I do have to add that I currently have my mail filtering disabled because I encountered a problem after I upgraded to 8.2.1. For some reason mail forwards that were being filtered were not being forwarded on.

    I am still looking into this issue, as it I am sure it has something to do with the return values.
     
  16. Hal9000

    Hal9000 Guest

    0
     
    i got the thing working in the meanwhile, in combination with horde ingo etc...
    BUT i just discovered that mailgroups don't work properly with the modified psa-spamc file, the mails don't get delivered to the mail groups (if spam checking is enabled for them)! so something needs to be changed there, and hopefully it can be changed :) but i guess is that $QMAIL_LOCAL knows about mail groups, whereas maildrop doesnt :(
     
  17. stumyp

    stumyp Guest

    0
     
    How about adding this functionality to the PLESK distribution package ?
    and additionally, it would be very helpful if spamassassin will learn from messages placed to Spam folder manually by user.
    Thnaks
     
  18. stupidnic

    stupidnic Guest

    0
     
    Progress Update

    I have spent some time debugging the problems with mail groups (as mentioned by HAL) and mail forwards as I have noticed personally.

    The problem is how we are delivering the files. Since maildrop has its own delivery mechanism, we are by passing the .qmail files for the user. This is where Plesk stores the mailgroups, and mail forwards for the users.... so when we by pass these .qmail files (since maildrop doesn't read them) we are by passing those additional delivery commands.

    How do you fix it?

    Well you have two options.

    Option 1: Turn off spam filtering for any mailgroup or mail forwarding account.
    Pros: Makes the delivery work as intended
    Cons: No spam filtering

    Option 2: Parse the .qmail files in maildrop and do additional logic based on the information in those files.
    Pros: Will cause delivery to work as intended
    Cons: More work for me to develop (code to parse the .qmail files properly)

    For now I am going with option 1, as I have enough control over my users to turn off spam filtering for those users that managed to turn it on for a mail forward/mail group.

    I am also actively working on getting something in place to handle the proper delivery options as defined in the .qmail file. I am sort of torn because I know there are qmail mail handler plugins that should be able to handle this sort of stuff in the Plesk API, I just can't seem to find the correct information in the API to tell me how to access it. The information is vauge at best, and down right obfuscated in other areas. <paranoia> It is almost like they don't want somebody to develop a plugin that would circumvent their SpamAssassin plugin, even though most of the functionality is there to do it.</paranoia>

    Anyways... that is where we are for now. I will update more when I have something more to share.
     
  19. stumyp

    stumyp Guest

    0
     
    Thanks, stupidnic.
    I've implented your solution with Plesk 8.3 and it works.
     
  20. stupidnic

    stupidnic Guest

    0
     
    Quick Work Around for Mail Forwards and Mail Groups

    Okay. I have a rather quick and dirty work around for the mail groups and mail forwards problem.

    Here is the modifications that you need to make to the psa-spamc file:

    On line 28 of the file add the following:
    Code:
        FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
        if [ $FORWARD -ne 0 ]; then
            "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
        else
            /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
            retval=0
        fi
    
    So the entire section should read:

    Code:
    if [ $check -ne 1 -o "$1" != "reject" ]; then
        FORWARD=`/bin/grep "^&" $HOME/.qmail | /usr/bin/wc -l`
        if [ $FORWARD -ne 0 ]; then
            "$QMAIL_LOCAL" -- "$USER" "$HOME" "$LOCAL" "" "" "$HOST" "$SENDER" "$DEFAULT" < "$tmpfile"
        else
            /usr/bin/maildrop /etc/mailfilter < "$tmpfile"
            retval=0
        fi
    else
        retval=99
    fi
    
    Basically this is handing the delivery off to qmail-local if it detects any line in mailboxes .qmail file that begins with & (dot-qmail parlance for delivery to another mailbox). It fixes the problem with delivery to mail groups and mail forwards, and still filters the messages if they are to be delivered locally.

    Just make sure that the paths to your grep and wc binaries are correct.

    I could add some additional logic to it to detect more things, but at this point and time, it works well enough, so I am going to leave it at that. I will edit the script I posted on the first page to reflect these changes as well, just incase nobody gets to the second page of the thread.

    Take care!
     
Loading...