Question Possible to use DNSBLs for moving identified email as spam to spam folder (not blocking)?

[tramvainqueur]

Server operating system version

Ubuntu 22.04 x86_64

Plesk version and microupdate number

Plesk Obsidian 18.0.62.2

Is it possible to use one or more DNSBLs, which does not block listed email as spam but having it moving to spam folder solely?

So I and the other users do not have trust to the dnsbl-creators fully and we all can look from time to time if a ham was marked falsely as spam. This would be perfect - so I would not have the fear of sometimes blocking an important email without knowing this -, but I do not know if spamassassin or plesk can make this possible. Is it? And if yes, an advice or solution how to implement this?

 

[danami]

@tramvainqueur Yes this is possible. Take a look at the "Enabling DNSBLs at the content filter level" section from the KB article below. The article was written for our Warden product but it should work the same for the default spamassassin that Plesk provides:

How can I add custom DNSBLs for use in Warden Anti-spam and Virus Protection? - Knowledgebase - Danami

 

[tramvainqueur]

@tramvainqueur Yes this is possible. Take a look at the "Enabling DNSBLs at the content filter level" section from the KB article below. The article was written for our Warden product but it should work the same for the default spamassassin that Plesk provides:

How can I add custom DNSBLs for use in Warden Anti-spam and Virus Protection? - Knowledgebase - Danami

Ok, I tried to implement changes by means of your suggestions and now I have to wait some daysto see if my change works as expected. Btw. I also implemeted because of a suggestion by another site the change of the score of a dnsbl used in spamassassin because several newsletters from the well known german news site get because of spamhaus a much to high score and in several real spams I did not find this dnsbl helping to identify it. It was sad becase I had to set the required score to 2.6 only for this german newsletters. Else the requiring score 0.7 helped to eliminate almost all spams.

If it works as expected, I will write down in net post the details what I have done. It would be great, if Plesk would make it possible to add the possibility to add different dnsbl, dnswl and other or to change scores of some lists and not to block solely.

Btw. @danami : For the advanced edit of spamassassin you recommend to edit /etc/mail/spamassassin/local.cf. But there is a probem with some systems (one type: systems using Plesk). That is a reason why Plesk makes me sometimes some headaches. After updates of Plesk and/or some things on the server, this local.cf can be overwritten (thankfully a warning in the text file itself advices to not edit this file manually). So it is better to make and edit /etc/mail/spamassassin/custom.cf or whatever name is better, so this file is permanent. I hope this is in conformance to Plesk, sufficient and does not require to include additional directive(s) in another place(s). Several pages I found think this is sufficient. At least command 'spamassassin --lint -D all 2>&1 | less' tells that the new cf-file seems to be read.

I hope all I changed is good, but as I am not sure, I wait and will analyze some days later some spams, so I can ensure at least empirically that this shall work now and in future. If I find in some spam-email headers my own directives, then I know it worked and I will tell here what I have done to achieve this.

Btw. @danami & @Kaspar@Plesk & @pleskpanel : As read on several sites for spamassassin ist shall be better to use bb.barracudacentral.org instead of b.barracudacentral.org. Plesk uses spamassassin, so I think this is more appropriate.

 

[tramvainqueur]

[...] Btw. I also implemeted because of a suggestion by another site the change of the score of a dnsbl used in spamassassin because several newsletters from the well known german news site get because of spamhaus a much to high score and in several real spams I did not find this dnsbl helping to identify it.[...]

Addendum: Interesting in this case someone can change the scores of ratings which seem to come from DNSBL's, this showed me that spamassassin has per default already some DNSBL's configured (they do not block, just move to spam folder). That is why spamassassin at first without manually configuring it works relatively good (not perfect, but not bad either).

After a half day some spams & hams indicate that my implementation seems to work. But I will test it more to ensure good working and then tell what I have done. For example to edit local.cf is bad as this is not permanent for a server with Parallels Plesk. It is sufficient to create a file with endning '.cf' like 'custom.cf'. No include is needed. Source codes of spams showed me that my implementation of directives with DNSBL's for moving are obeyed.

 

[danami]

@tramvainqueur Plesk does not overwrite /etc/mail/spamassassin/local.cf.

 

[tramvainqueur]

@tramvainqueur Plesk does not overwrite /etc/mail/spamassassin/local.cf.

Good to know. Is this warning useless? Or is the reason a distribution upgrade of Ubuntu/Debian overwrites this?

But nevertheless imho it is cleaner using a separated config file for custom changes, or?

 

[tramvainqueur]

As promised I try to report what I have done and worked, but first some preliminary informations.

I am owner of several personal domains with different TLD's. One has been registered in the year 2000 and the mail-address became in about 2015 round about 180 spams per day (yes, in the year 2000 spams were no problem in germany, so I wrote the address several times in the clear net). But such things like DMARC or components had not been configured and did not exist in these early times.

Some data about the email daemon on the server I will talk: It runs the newest Parallels Plesk Management System. There I installed this Plesk Secure Mail Extension (for integration with spamassassin), but I have not an abonnement activ (is a 'little' private server with some customers) and use just only the free features of this extension (one reason: it installs amavis and makes automatically a connection with spamassassin). This is why some network configurations for email are not the same and perhaps it is the reason, why spamassassin did not learn from new spams until I integrated this manually as cron job, but here it does not matter anything.

My realization with help of @danami what helped "my dream" to come true, that DNSBL's do not just block possible spams but move them to the spam folder, so anyone can control if spam filter worked and are able to move false positives back to an ham folder/inbox or folder in inbox without spams (but from my experience with good configuration there is never a false positive in the spam folder ... only sometimes spam in inbox but not much):

I realized enabling the so called DNSBL's (normally wrong word as 'BL' means 'block list' but with spamassassin possible spams are not blocked ... just moved to spam folder and perhaps learned by the spam filter ...) at the content filter level what website calls 'command line interface'. Yes, it is not that easy like with Plesks Server Management per website, but it gives you really much more options, so one may even deactivate a default 'DNSBL' (for example I reduced the score of 'URIBL_DBL_SPAM' of spamhaus.org, because it produced really many false positives, although some test sites allege it produces no false positives but I think it is because of bad test data ... I am sure their test data does not include mails in different anguages ... my false positives arre all not english (german, spanish, french, italian, portuguese and more to be detailed) ...). That is why I heavily recommend to do this per shell on the server in a terminal (for windows users called 'command line interface' although not powerfull ... microsoft power shell is more complex and better than the often used shell bash in the POSIX systems like unix, linux and so on, but that complicated that almost no one uses this ) directly or with SSH, if possible and not damned to use only the Plesk Server Management Interface.

Step 1: Login with SSH in to the server (I assume server runs with a linux or unix distribution ... with windows servers I do not have experience).

For linux, unix or posix desktops reaching linux, unix or posix servers for example following in terminal (for windows users it shall be similar with putty possible):

ssh [email protected]

Step 2: Go to folder '/etc/mail/spamassassin/'
Step 3: Create an empty text file with the ending '.cf' like 'custom.cf' to add your own rules. I do not recomend to edit 'local.cf' a in this file is written

ATTENTION!
#
#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

. @danami says this file is not ovewritten by plesk, but I assume this text is not senseless. I think with an update or upgrade of spamassassin or the distribution itself this file will be overwritten and the own changes will be lost. By the way it is imho cleaner to separate custom modifications from other modifiers.

Step 5: Write in 'custom.cf' DNSBL's with text like

header          C_RCVD_IN_BARRACUDACENTRAL  eval:check_rbl('barracudacentral-lastexternal','bb.barracudacentral.org.')
describe        C_RCVD_IN_BARRACUDACENTRAL  Listed in bb.barracudacentral.org DNSBL
tflags          C_RCVD_IN_BARRACUDACENTRAL  net
score           C_RCVD_IN_BARRACUDACENTRAL  0 # please adjust the score value

.

Little explanations: First column is a must (not always the rows as when only a score shall be different ... later more ...). From statistics of some dnsbl's with spamassassin description in details I take the original names like 'RCVD_IN_BARRACUDACENTRAL' and can be modified without problems (I add 'C_' at the beginning for indicating 'custom' and this change has to happen in every row at the same column. In the last row one can set the wished score (0 deactivates this 'dnsbl').

Step 4: Execute in the shell '/usr/bin/systemctl restart amavis' on ubuntu server with amavis installed and configured (on other sever something similar that restart spamassassin). Now the new '.cf'-file is respected by spamassassin and no special includes in other config files are needed. With 'spamassassin --lint' one can test in forwars if rule is accepted, correct the rule if needed and the restart amavis or spamassassin.

Helpful for emails with foreign language so spamassassin has much less or no false positives: Just add

score URIBL_DBL_SPAM 1

(or lower score like 0.5 or higher score like 1.5) to 'custom.cf'

 

[tramvainqueur]

addendum:

I have reread the posting and found some mistakes. But as I see, they shall not hinder to understand my realization. As I had to end earlier as thought, I missed to correct some things and I did not write what the result is.

Result is generally spoken good, so without paying 'email experts' one can achieve that only every pair of days a spam appears in the inbox and no ham/false positive appears in the spam folder.. And the best is, no full trust in these 'dnsbl's is needed, as with spamassassin no email is blocked without informing someone.



Details (now since one week - spamassassin not to my wish perfect configured yet ... will take perhaps some weeks for testing new 'dnsbl's and fine adjustments ...):

usecase 1 (me): has many email addresses on several own domains for over a decade; trys to be always correct (useless subscripted emails never moved to spam but derigestered or simply deleted (else spamassassin will move many legitim emails to the spam folder); real spam moved to spam folder and learned by spamasssassin; on many mailing lists and registered many news letters (some because of a good reason I will not tell here are very old and do not even remember all places); has several account with intel, oracle, lenovo, ibm, affinity, maple, mathworks, matlab, several shops and much more

usecase 2 (account from another one where I am allowed to have full rights and I am their private sys admin): uses account conservatively (account on server newly, therefore spamassassin learned from spams only some years); registered address on some news letters and mailing lists; has some accounts in several shops

usecase 3 (account from another one where I am allowed to have full rights and I am their private sys admin): uses account very freely and has no fear to announce it publicly (account on server newly, too, therefore spamassassin learned from spams only some years); registered address on many news letters and maling lists; has several accounts (most of them not known by the user itself ... ... ok, is not tech affine and has a little aversity against these things, but trys to use them ...); I think address is posted in the clear net on several sites

Results:

usecase 1: round about after a pair of days a spam appears in inbox and no false positive in spam folder (although with all email accounts I get more about 80 spams on main address and about 120 spams all other addresses counted together, but most spam is filtered)

usecase 2: round about 1-2 spams per day (there are days with 3 and more spams, but also many days with 1 and some even with 0 spams)

usecase 3: round about 2-5 spams per day (here back again some days more and on some days less ... I think days with 0 spams I did not find yet ...)

Indeed usecase 3 is my little problem child yet, but the 'dnsbl' is not fully configured yet and surely more fine adjustments are needed. And perhaps some spam fighting technics are easily addable. Sadly the user itself is a little problem, too. Some spams in inbox have been learned as ham accidentaly, as user does not strictly sort out spam from inbox. This user thinks it should just work and does not try much to keep the inbox clean. But as this configuration type is relatively fresh, there is much hope, yet, that these problems will vanish or at least diminish significantly.

Now it is late and I have to go to sleep, so I hope this text has not too many mistakes which hinder to understand this post. If something is not understandable, then please ask me and I will try to explain it in another way.

 

[tramvainqueur]

Little update: The first 3 days indicate that my little updates in 'custom.cf' and score/'fine' adjustments have been already sufficient to achieve that usecase 3 and 2 are at least that good as usecase 1, or perhaps even better, but no complete week after the week with first configuration has past yet.

 

[tramvainqueur]

Results (which will vary/surely get better in the future if spammers are not faster getting better):

usecase 1: Now after third week I get about 0.28 spams per week in inbox. About 0.28 hams appear in the spam folder, but spamassassin learns with my rules from time to time (so hams in spam folder are in next week not in spam folder anymore). Not perfect but I can live with it. With my adjustments I reduced the false positives almost completely - before I had several false positives per day(!). How good spamassassin works seems to depend from user. Although I have about 5 'private' (also owner admin-c, but not used solely privately!) domains and use about 7 email accounts (and much more with aliases ... about 70 as I see ... yes, a lot are not actively used and many are for administrative reasons created, but about a third I use them actively ...) I am the usecase with the least spams in inbox and the least false positives. The other usecases have only one account, one domain and only some alias (mostly administrative reasons).

usecase 2: It becomes now about 3 spams per week in the inbox. It became better because of my modifications (spamassassin sets per default scores some lists and to email properties and it seems to be empirically tested with a spam database, but this database seems not to have not much international emails).

uscase 3: It becomes 1-2 spams per day. False positves are about 0.28-5 per day. Why 0? From usecase 3 I naturally do not know in which mailing lists, newletters or organizations it registers. Analyzing the so called spam some appear superficially spam but further analysis told me they are indeed hams. But some of the legal emails contain only ads which do not have to do much with the organization. So for these emails I do not costumize the configuration for spamassassin as they are not worth it. Usecase 3 can contact me and tell me if they are worth indeed but this did not happen yet (I even asked usecase 3 and it told me they are not important or needed). At least looking in spam folder can help. And I am sure if some of these strange false positives are moved to inbox, in near futere they will not appear again as false positives.

Lastly, result is not perfect (but 'almost'!) and has potentials to get better yet but it is very much better now (at least for international users). Phantastical is now that you can 'play' with spamassassin configuration and one does not have to have fear some important emails are blocked radically without being informed. If false positves happen they appear in spam folder now and can be retrieved if in less than 30 days this spam folder is examined. Back again I thank @danami for poking me to this method (which I extended a bit) because now I could reduce really many spams. I hope the team of Parallels Plesk makes it possible to change 'local.cf' and to be able to edit with a custom CF-file easily within the server management web interface.

Some usernames to hopefully become additionnal attention for this: @admin @Aleksei Stepanov @Bitpalast @alapshin

 

[tramvainqueur]

Little update:

Perhaps because of my additionnal cronjobs with sa-learn in folders '.spam' with command '--spam' and in folder '.archive' with command '--ham' spamassassin seemed to have learned fast (and the usecases followed my advice moving false positives to '.archive'). Last week I got even 0 spam in inbox and 1 ham in spam folder, whereas this unique false positive is from a not important mailing list (and perhaps spamassassin learns in future these shall not go to spam).

This was usecase 1. Usecase 2 did not change much. Usecase 3 got better, too. There seem to be some false positives in '.spam', whereas from some of the some I do not know if they are spam, indeed, because I do not know in which mailing lists usecase 3 registered or typical spam that alleges to be from a mailing list and the not working 'remove link' just tells the spammer this email box lives.

The possibility of avoiding all blocking and then being able of fine tuning the configuration of spamassassin, was a really great game changer!

 

[tramvainqueur]

Little and surely last update, if no one wants more details how to implement what:

Usecase 1 has some hams in spam folder, but moving them to a folder or subfolder of archive, where in cron job I set spamassassin shall learn all emails as ham from there, is enough so spamassassin the next or in two days does not move these special hams to the spam folder. Usecase 2 seems to be even better than usecase 1, now. But one has to take in account, person of usecase 1 does not get emails in many different languages. Usecase 3 has become much better. No, it is not my box and therefore I do not have full control, but it seems to be now that good as usecase 1. Usecase 3 gets also letters in different languages. So email account of usecase 3 is no problem child anymore. I thank for the advice of @danami and many thanks to the developers of spamassassin, which made my dream come almost true of 0 spam and 0 false positives.

 

[tramvainqueur]

I begged Parallels Plesk developers to implement to be able to modify the CF-Files of spamassassin in the plesk web server managment panel without having to access with SSH the server and add a conforming file 'by hand'. Please vote on my beg on this site. Yes, I do not need this, but for others with less knowledge about servers this would be surely helpful (and it would be a positive argument for using and buying Parallels Plesk).

 

[Raul A.]

You could write a Plesk extension that will do just that. However the rules will apply to all email addresses hosted on your Plesk server, not just a specific email address.
What changes do you want to make to the CF-Files of Spamassassin?

 

[tramvainqueur]

You could write a Plesk extension that will do just that. However the rules will apply to all email addresses hosted on your Plesk server, not just a specific email address.
What changes do you want to make to the CF-Files of Spamassassin?

This is a bit sad if the rules in CF-files can not be restricted to explicetely defined email addresses, but here I only wanted that these rules apply to all email addresses. I use the host edition and I am naturally an administrator of this hosting server. And I am happy to being able to use the host edition of Parallels Plesk. That is why I hope the developers can add such a setting at least for the other administrators/roots of their server. If CF-files restricted for some users, domains and/or email addresses can be implemented, this would be great, but I think this not really necessary IMHO (but perhaps I do not know all usecases).

What changes I wish?

Background: When esperimenting to set the spam filter better, I was a bit astonished that without rules written in Plesk web panel a lot of spams were sorted out. In source code of these spams I saw, that spamassassin uses some DNSBL's that I did not define. It was awful that some emails in non-english languages (of private persons or newsletters) were moved always to the spam folder (many false positives), although about 10-20 real spams I had in inbox yet (level of spam filter was not defined hard). So if I wanted less spam in inbox, I had to define spamassassin levels harder. Then I have less spam but much more false positives. This problem seemed not to be solvable. Then because of @danami and other sites I learned one can change the scores of the used DNSBL's (and important: how to set that the possible spam never are blocked copletely but moved always to the spam folder so afterwards I can rescue the false positives). On the other hand on spamassassins official site I read it already uses some DNSBL's as help for detecting spams. So I saw there seems to be a possible solution with changing the score system of spamassassin. The developers of spamassassin surely have developed a very good score system, but for their bad email data base with spams and hams. For example with english emails I had almost never problems. But with emails in german, spanish, italian, french and other languages I got several false positives (and really annoying was, as a daily newsletter from a great non-english site was moved always to the spam folder although I tried several times that spamassassin learns that these emails are ham). So for non-english emails the score system of spamassassin is not absolutely bad, but bad enough so their score system can be made really much better.

A bit information from what should be changed (but I tell not everything because it would be huge and my system is surely not perfect ... perhaps for some even bad, as I can not know all possible usecases): Many false positives produces Zen with 'URIBL_DBL_SPAM' (predefined in spamassassin!). In CF-file I made the score about 50% lower. 'DEAR_SOMETHING' gave me almost always false positives, so I set his score to 0. Of 'RCVD_IN_BARRACUDACENTRAL' I reduced its score by 1, so the great newsletters are no more false positives but for some real spams it keeps helping a bit in detection. In the mean time I modified the scores from about 21 DNSBL's and from 12 predefined rules in spamassassin. My system reduces drastically the amount of spams to about some spams in the month and I have only some or even 0 false positive in the month. So my system ist not perfect naturally (well, sadly this will be a never ending race between spammers and defenders, as long the email system exists), but it got much better. Before I personally got without spamfilter about 90-120 spams (yes! Ninety till hundredandtwenty!) per day(! not per month!).

Well, if it is not very complex, I could write an extension. But I I fear this requires much time which I do not have at the moment. But such a setting is fundamental and Plesk is comercially sold. So I kindly point to a possibility how to higher the reputation of Plesk which surely leads to more sales of Plesk. And this would a really little change for developers already envolved in the development. It is not OpenSource, where a noticably amount of persons shout "Do it yourself! Or even help the community by changing the OpenSource!". Myself lastly I do not need this feature as I learned it in an hard way how to solve this problem, but other customers will be surely pleased if with less learning curve they can achieve the same (... well, they will have to get a bit of knowledge how this works, but hey, why are they admins, if they do not want tolearn technical aspects?). If Plesk developers want to ignore my helpful suggestion, then it is their fault.