Syntax for Greylistingfilter (Blac-/Whitelist)

[Flachzange]

Syntax for Greylistingfilter (Black-/Whitelist)

Hello,

I am using the "new" PLESK 9.2.1 greylisting feature since two weeks now. But I am not able to figure out which syntax is used to define the Black-/Whitelist. On the first sight it seems to be Regular Expressions as a default entry is for example

dynamic|ppp|dyn-ip|dial-up|dsl

So I added an entry to filter out all senders without a fqdn, only having an IP address. But what ever expression I try, they do not work. The expressions itself are ok, they have been valided before.

So I hope there is someone here, who can help me with this issue.

Thx
Christoph


Edit:

On of the default entries in the db is:

.*[0-9][0-9]\.[0-9][0-9]\.[0-9][0-9].*

But when I receive spam from a server with the IP Address: 84.79.151.165 (no fqdn)

This Mail is not rejected, although the pattern matches.

The address 194.154.72.131 is correctly rejected using the pattern above

 

[Denis Sirochenko]

Hello,

Take a look at Parallels Plesk Panel 9.2 for Linux/Unix Command Line Reference:

Greylisting: grey_listing Utility
http://download1.parallels.com/Plesk/PPP9/Doc/en-US/plesk-9.2-unix-cli/63188.htm

Regards,
Denis

 

[Flachzange]

Hi Denis,

thanks for your reply. I am actually not using the cmdline tool and I haven't used it before. Instead I am using the Greylisting Manager from http://www.haggybear.de/, which actually seems to use the cmdline anyway and makes the entire greylisting management very handy. But nevertheless, the reference description of it does not help me with my problem. Today I went through my logs and found another example which demonstrates the problem I have:

I have the following server wide blacklisted domain pattern:

*dsl*|*dynamic*

Then I found two examples in the log file

1)The pattern is applied

Jul 12 00:14:27 hostname postfix/smtpd[11532]: connect from acyh104.neoplus.[B]adsl[/B].tpnet.pl[83.11.191.104]
Jul 12 00:14:28 hostname postfix/smtpd[11540]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
Jul 12 00:14:28 hostname postfix/smtpd[11540]: connect from localhost.localdomain[127.0.0.1]
Jul 12 00:14:28 hostname postfix/smtpd[11532]: NOQUEUE: client=acyh104.neoplus.adsl.tpnet.pl[83.11.191.104]
Jul 12 00:14:28 hostname postfix/smtpd[11540]: 230C514F1E54: client=acyh104.neoplus.adsl.tpnet.pl[83.11.191.104]:2490
Jul 12 00:14:28 hostname before-queue[11537]: check handlers for addr: [email protected] 
Jul 12 00:14:28 hostname before-queue[11537]: check handlers for addr: [email protected]
Jul 12 00:14:28 hostname before-queue[11537]: Processing handlers... 
Jul 12 00:14:28 hostname before-remote[11539]: check handlers for addr: [email protected] 
Jul 12 00:14:28 hostname before-remote[11539]: check handlers for addr: [email protected]
Jul 12 00:14:28 hostname before-remote[11539]: Processing handlers... 
Jul 12 00:14:28 hostname before-queue[11537]: hook_dir = '/opt/psa/handlers/before-queue'
Jul 12 00:14:28 hostname before-queue[11537]: recipient[3] = [email protected]'
Jul 12 00:14:28 hostname before-queue[11537]: handlers dir = '/opt/psa/handlers/before-queue/recipient/[email protected]'
Jul 12 00:14:28 hostname before-queue[11537]: call_handlers: call executable = '/opt/psa/handlers/info/05-grey-6It4LH/executable'
[B]Jul 12 00:14:28 hostname greylisting filter[11542]: list type: black, from: acyh104.neoplus.[B]adsl[/B].tpnet.pl, match string: .*dsl.*|.*dynamic.*
Jul 12 00:14:28 hostname before-queue[11537]: handlers_stderr: REJECT[/B]

2)The pattern is NOT applied

Jul 12 01:48:06 hostname postfix/smtpd[11588]: connect from r190-135-9-208.dialup.[B]adsl[/B].anteldata.net.uy[190.135.9.208]
Jul 12 01:48:07 hostname postfix/smtpd[11614]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
Jul 12 01:48:07 hostname postfix/smtpd[11614]: connect from localhost.localdomain[127.0.0.1]
Jul 12 01:48:07 hostname postfix/smtpd[11588]: NOQUEUE: client=r190-135-9-208.dialup.adsl.anteldata.net.uy[190.135.9.208]
Jul 12 01:48:07 hostname postfix/smtpd[11614]: B3B9E14F1E54: client=r190-135-9-208.dialup.adsl.anteldata.net.uy[190.135.9.208]:2047
Jul 12 01:48:08 hostname before-remote[11613]: check handlers for addr: [email protected] 
Jul 12 01:48:08 hostname before-remote[11613]: check handlers for addr:  [email protected]
Jul 12 01:48:08 hostname before-remote[11613]: Processing handlers... 
Jul 12 01:48:08 hostname before-queue[11611]: check handlers for addr: [email protected] 
Jul 12 01:48:08 hostname before-queue[11611]: check handlers for addr: [email protected] 
Jul 12 01:48:08 hostname before-queue[11611]: Processing handlers... 
Jul 12 01:48:08 hostname before-queue[11611]: hook_dir = '/opt/psa/handlers/before-queue'
Jul 12 01:48:08 hostname before-queue[11611]: recipient[3] = [email protected]'
Jul 12 01:48:08 hostname before-queue[11611]: handlers dir = '/opt/psa/handlers/before-queue/recipient/[email protected]'
Jul 12 01:48:08 hostname before-queue[11611]: call_handlers: call executable = '/opt/psa/handlers/info/05-grey-6It4LH/executable'
Jul 12 01:48:08 hostname greylisting filter[11616]: Starting greylisting filter... 
Jul 12 01:48:08 hostname before-queue[11611]: handlers_stderr: DEFER

So my question is: Why does the greylisting filter behave like it behaves?

Any suggestion is appreciated..

Regards,
Christoph

 

[Flachzange]

hmm...does nobody has encountered the same problem? Or is just nobody using the plesk internal greylisting filter?

it's a bit frustating

Christoph

 

[Kevin Troendle]

Please clarify Plesk

I agree, the behavior here needs to be explained by Parallels.

The fact that is a hidden behavior is not acceptable either.

 

[Flachzange]

Please clarify Plesk

I have sent two detailed bug reports to the Plesk bug report address, but I have never got any answer from them.

 

[trialotto]

Strange indeed.

However, some thoughts (rather being questions):

- the servers and domains are different. Are those domains/servers having different functions and settings, f.e. are they both pure relay servers or not?
- Note the format of one the server adresses is simply the reverse DNS of the IP adress!
- in both cases the same handler is used: why do you not check the handler? (and improve if necessary, since custom mail handlers are allowed)
- in both cases the result is similar: defer and reject, it is not a great difference
- why are you using the haggybear thing? It can be easy, but the program can also cause translation problems....have you analyzed whether the input in the haggybear thing is resulting in the same input in the plesk grey listing filter?

In my opinion, the behaviour can be explained by the fact that one of the servers is a reverse DNS.

However, that does not resolve your problem yet.

Can you give me first some answers to my thoughts/questions?

It is very likely that your answers will yield the conclusion that it all has to do with the reverse DNS adress. If true, then plesk really has to tweek the standard handlers for the greylisting filter.

NOTE: your answers can help isolating the problem, a problem in which i am very interested: after all, if greylisting is not working properly for reverse DNS adressesses, then a potential risk exists

 

[Flachzange]

Hi trialotto,

thanks for your answer. I will try to answer your questions/statements as good as possible.

- the servers and domains are different. Are those domains/servers having different functions and settings, f.e. are they both pure relay servers or not?

If you mean for example that acyh104.neoplus.adsl.tpnet.pl is different to [email protected]. This due to the fact that spammers fake the sender address, thats why the plesk greylisting filter uses reverse dns instead of domain names in the mail address.

- Note the format of one the server adresses is simply the reverse DNS of the IP adress!

of course, that is what plesk is using for the greylisting black- and whitelists

- in both cases the same handler is used: why do you not check the handler? (and improve if necessary, since custom mail handlers are allowed)

Unfortunately I don't have any experience with custom mail handlers and debugging them

- in both cases the result is similar: defer and reject, it is not a great difference

This seems similar but in fact it is a huge difference. While rejecting blocks an incoming mail request immediately without giving the sender any second chance, deferring means the greylisting mechanism, i.e. a second attempt from the same reverse DNS to the same mail receipient will lead to a SKIP which is equivalent to let the mail pass.

- why are you using the haggybear thing? It can be easy, but the program can also cause translation problems....have you analyzed whether the input in the haggybear thing is resulting in the same input in the plesk grey listing filter?

pure convenience. Plesk does not ship with any graphical management tool for greylisting. And the Haggybear Greylisting Manager does a good job. Of course I have checked the black- and whitelisting entries directly in the database to make sure that the greylisting manager is not the cause of the problem.

In my opinion, the behaviour can be explained by the fact that one of the servers is a reverse DNS.

as stated above: greylisting only uses reverse dns addresses and both of the above server addresses are the reverse DNS names.


So I hope it became a bit clearer by now and thanks again for your answer.

Christoph

 

[Flachzange]

One addition: the documentation of the cmd line utility (see second post in this thread above) talks about list entries for "mail addresses" and for "domains". As far as I understood it correctly "domains" refers to reverse dns entries. (Please correct if I am wrong, but nothing else makes sense for me) So it is actually possible to filter by reverse dns entries and concrete mail addresses. But the latter ist quite useless in most of the cases, as spammers vary the sender address in many ways or even use the receipent as sender.

 

[trialotto]

Greylisting behaviour

The answer to your original question seems to be more straightforward then we thought (see 3).

Some of the typical greylisting behaviour can be explained by discussing the way greylisting works.

Greylisting is using "triplets" which contain the IP adresses of the sending server and the sender and receiver mail adresses. Unknown triplets are not being delivered during a short period of time (can be configured) and a normal, legitimate mail server would try again.

The theory is that spammers do not have patience and do not try again.

Hence, in response to your replies/questions, the following can be stated:

1 - Use of concrete mail addresses
In fact, one uses the sender's mail address and that is part of the triplet. Putting the sender on a white or black list allows mail to get through or not.

The concrete mail addresses can offcourse be using wildcards or regexp. A concrete mail adresses in the format of *@<domainName>.tld blocks or allows the corresponding mails (depending on being on the white or black list).

So that is indeed a feasible option for spam mail coming from specific adresses. However, i do agree: it is quite cumbersome to declare expressions that capture a lot of spam, since spammers use huge variation in the spam mail adresses.

Impersonating the recipient however, that is something that should not be possible in decent qmail configurations. If possible, it is not a greylisting failure, but a failure in setting up proper configuration.

2 - Greylisting and reverse dns
Greylisting does not necessarily refer to reverse dns entries, when talking about domains. In fact, every type of expression (that can be resolved to an IP address) can be used in the greylisting filter.

However, there seemed to be an issue with reverse dns in plesk. To my best knowledge, this should not interfere with greylisting working properly.

3 - Behaviour and difference in REJECT and DEFER
In the normal greylisting process, the filter would be analyzing the "triplet" and compares that to data in a database (or stored otherwise). The before mentioned data at least consists of the white and black lists.

When the triplet is unknown, the mail should be deferred (and retried later). When the triplet is known to be spam, the mail should be REJECTED (permanently).

Since greylisting is a system storing data on triplets, this would be explaining your maillog. Somewhere in the process, your mail system has learned that one of them should be rejected directly.

NOTE: deferring is not resulting in SKIP of messages when the sending machine retries. If it does in your case (can be, but should not be), just re-install your mail server or recreate the mail handlers (handler creation should be sufficient).

NOTE: deferring mails is dangerous for badly configured mail servers. A re-install or fine-tuning of configuration is at least required if multiple mails are being deferred. One of the starting points should be the configuration of the greylisting filter (penalties, timing intervals, better black- and whitelists etc).

4 - Question on my behalf (!)
If i am not mistaken, two things can be causing the exact maillog you have:

1 - You restarted or re-installed the mail server (or at least, you did a rebuild of mail handlers) between 00:14 and 1:48,

OR

2 - somewhere in your maillog there should be similar entries to those of the case of [email protected]. These entries should be present before 00:14 and the message is very likely of the type DEFER.

My question: which of both is the case?

If neither is the case, then there really is an issue that is more profound and potentially severe (a handler not working properly, or a reverse dns or IP related problem).

 

[Flachzange]

Greylisting behaviour Part 1

The answer to your original question seems to be more straightforward then we thought (see 3).

Actually not, as my original question referred to the syntax for entries of the back and whitelist, which is not clear yet.

Regarding to the command line tool documentation "some kind of" patterns (regexp?) are allowed for mail addresses, but only domain names are allowed for domains. BUT plesk ships with default entries for the white and blacklist. The greylisting database /var/lib/plesk/mail/greylist/settings.db contains four tables:

1) local domains (empty in my case)
2) mailnames (refers to concrete mail addresses/patterns; shared with spamassassin)
3) remote_domains (refers to entire domains
4) settings (contains greylisting settings: penalties and expiration settings)

An excerpt from settings_db and table remote_domains:

Whitelist plesk default entries:
*google.com
*mail.ru
*parallels.com
*rambler.ru
*yahoo.com
*yandex.ru

Blacklist plesk default entries:
*[0-9][0-9]-[0-9][0-9]-[0-9][0-9]*
*[0-9][0-9].[0-9][0-9].[0-9][0-9]*
*[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9]*
*[0-9][0-9][0-9].[0-9][0-9][0-9].[0-9[0-9]][0-9]*
dsl|pool|broadband|hsd
dynamic|static|ppp|dyn-ip|dial-up

As one can see, does the remote_domains also contain some kind of patterns by default. (As above stated does the command line tool help only document the use of patterns in case of mail addresses and not for domains.)

And as one can also see are patterns included for ip addresses, which do not offer reverse DNS resolution, e.g.
*[0-9][0-9].[0-9][0-9].[0-9][0-9]* or *[0-9][0-9][0-9].[0-9][0-9][0-9].[0-9[0-9]][0-9]*

Some of the typical greylisting behaviour can be explained by discussing the way greylisting works.

Greylisting is using "triplets" which contain the IP adresses of the sending server and the sender and receiver mail adresses. Unknown triplets are not being delivered during a short period of time (can be configured) and a normal, legitimate mail server would try again.

The theory is that spammers do not have patience and do not try again.

Hence, in response to your replies/questions, the following can be stated:

1 - Use of concrete mail addresses
In fact, one uses the sender's mail address and that is part of the triplet. Putting the sender on a white or black list allows mail to get through or not.

The concrete mail addresses can offcourse be using wildcards or regexp. A concrete mail adresses in the format of *@<domainName>.tld blocks or allows the corresponding mails (depending on being on the white or black list).

So that is indeed a feasible option for spam mail coming from specific adresses. However, i do agree: it is quite cumbersome to declare expressions that capture a lot of spam, since spammers use huge variation in the spam mail adresses.

Impersonating the recipient however, that is something that should not be possible in decent qmail configurations. If possible, it is not a greylisting failure, but a failure in setting up proper configuration.

clear up to this point...

2 - Greylisting and reverse dns
Greylisting does not necessarily refer to reverse dns entries, when talking about domains. In fact, every type of expression (that can be resolved to an IP address) can be used in the greylisting filter.

What else can be resolved to an ip address than a full qualified domain or just nothing (e.g. when having dial-up connections without any reverse dns entry)? I understand it like this:
1) An external Mail Server connects to my qmail with its IP www.xxx.yyy.zzz and wants to send a mail to a local address.
2) My local greylisting handler (or qmail) performs a reverse dns lookup for this IP
3) The greylisting handler checks if the result of the lookup is blacklisted or whitelisted
4a) In case of whitelisting the result is SKIP letting the mail pass. A very current log entry from this morning:

Oct 17 07:43:20 hostname greylisting filter[19914]: Starting greylisting filter... 
Oct 17 07:43:20 hostname greylisting filter[19914]: list type: white, from: mail.gmx.net, match string: mail\.gmx\.net
Oct 17 07:43:20 hostname qmail-queue-handlers[19913]: handlers_stderr: SKIP
Oct 17 07:43:20 hostname qmail-queue-handlers[19913]: call_handlers: SKIP during call '/opt/psa/handlers/info/05-grey-6It4LH/executable' handler

4b) In case of blacklisting the result is REJECT. Another current example from this morning:

Oct 17 10:40:55 hostname greylisting filter[19794]: list type: black, from: dsl.static859997228.ttnet.net.tr, match string: dynamic|ppp|dyn-ip|dial-up|dsl|dhcp
Oct 17 10:40:55 hostname qmail-queue-handlers[19792]: handlers_stderr: REJECT
Oct 17 10:40:55 hostname qmail-queue-handlers[19792]: call_handlers: REJECT during call '/opt/psa/handlers/info/05-grey-6It4LH/executable' handler

4c) In case of greylisting (no matching entries for White-/Blacklist) we have the two phase greylisting procedure. Also an example from today:
The first sending attempt resulting in a DEFER:

Oct 17 02:18:06 hostname relaylock: /var/qmail/bin/relaylock: mail from 87.248.114.198:42803 (web24301.mail.ird.yahoo.com)
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: Handlers Filter before-queue for qmail started ...
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: [email protected]
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: [email protected]
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: hook_dir = '/opt/psa/handlers/before-queue'
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: recipient[3] = '[email protected]'
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: handlers dir = '/opt/psa/handlers/before-queue/recipient/[email protected]'
Oct 17 02:18:06 hostname qmail-queue-handlers[14308]: call_handlers: call executable = '/opt/psa/handlers/info/05-grey-6It4LH/executable'
Oct 17 02:18:06 hostname greylisting filter[14309]: Starting greylisting filter... 
Oct 17 02:18:07 hostname qmail-queue-handlers[14308]: handlers_stderr: DEFER
Oct 17 02:18:07 hostname qmail-queue-handlers[14308]: call_handlers: DEFER during call '/opt/psa/handlers/info/05-grey-6It4LH/executable' handler

The second attempt resulting in a SKIP is letting the mail pass (if there no other restrictions like penalities or expired entries):

Oct 17 02:24:47 hostname relaylock: /var/qmail/bin/relaylock: mail from 87.248.114.198:47088 (web24301.mail.ird.yahoo.com)
Oct 17 02:24:47 hostname qmail-queue-handlers[17452]: Handlers Filter before-queue for qmail started ...
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: [email protected]
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: [email protected]
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: hook_dir = '/opt/psa/handlers/before-queue'
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: recipient[3] = [email protected]'
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: handlers dir = '/opt/psa/handlers/before-queue/recipient/[email protected]'
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: call_handlers: call executable = '/opt/psa/handlers/info/05-grey-6It4LH/executable'
Oct 17 02:24:48 hostname greylisting filter[17453]: Starting greylisting filter... 
Oct 17 02:24:48 hostname greylisting filter[17453]: Timeout finished
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: handlers_stderr: SKIP
Oct 17 02:24:48 hostname qmail-queue-handlers[17452]: call_handlers: SKIP during call '/opt/psa/handlers/info/05-grey-6It4LH/executable' handler

Maybe you could give me an example here if this differs from your point of view.

3 - Behaviour and difference in REJECT and DEFER
In the normal greylisting process, the filter would be analyzing the "triplet" and compares that to data in a database (or stored otherwise). The before mentioned data at least consists of the white and black lists.

When the triplet is unknown, the mail should be deferred (and retried later). When the triplet is known to be spam, the mail should be REJECTED (permanently).

NOTE: deferring is not resulting in SKIP of messages when the sending machine retries. If it does in your case (can be, but should not be), just re-install your mail server or recreate the mail handlers (handler creation should be sufficient).

please see my examples above

Since greylisting is a system storing data on triplets, this would be explaining your maillog. Somewhere in the process, your mail system has learned that one of them should be rejected directly.

no, the maillog explicitly says,:

Jul 12 00:14:28 hostname greylisting filter[11542]: [COLOR="Red"][B]list type: black, [/B][/COLOR]from: acyh104.neoplus.adsl.tpnet.pl, match string: .*dsl.*|.*dynamic.*

which means REJECT because of blacklisting

NOTE: deferring mails is dangerous for badly configured mail servers. A re-install or fine-tuning of configuration is at least required if multiple mails are being deferred. One of the starting points should be the configuration of the greylisting filter (penalties, timing intervals, better black- and whitelists etc).

sure, thats a general issue when using greylisting. This is the reason why I am studying logs and putting mail servers/domains on the whitelist which cause problems due to misconfiguration or just because they are using different different IPs with every sending attempt. But I dit not have missing mails until today, as really bad misconfigured servers are rare.

 

[Flachzange]

Greylisting Behavior Part 2

4 - Question on my behalf (!)
If i am not mistaken, two things can be causing the exact maillog you have:

1 - You restarted or re-installed the mail server (or at least, you did a rebuild of mail handlers) between 00:14 and 1:48,

OR

2 - somewhere in your maillog there should be similar entries to those of the case of [email protected]. These entries should be present before 00:14 and the message is very likely of the type DEFER.

My question: which of both is the case?

If neither is the case, then there really is an issue that is more profound and potentially severe (a handler not working properly, or a reverse dns or IP related problem).

It is neither the case as my original problem is different to the discussion we had now My original syntax related question was:

Why are some reverse DNS domain names backlisted while others are not. Concrete:
Why is

Jul 12 00:14:28 hostname postfix/smtpd[11532]: NOQUEUE: client=acyh104.neoplus.adsl.tpnet.pl[83.11.191.104]

rejected by blacklisting using the pattern (in table remote_domains in settings.db) :

.*dsl.*|.*dynamic.*

and why is

Jul 12 01:48:07 hostname postfix/smtpd[11588]: NOQUEUE:  client=r190-135-9-208.dialup.adsl.anteldata.net.uy[190.135.9.208]

not blacklisted by the same pattern.
As you may notice, have I switched from postfix to qmail in the meantime, but the problem is still persistent. So it is mailserver independent.

I am sorry for this large post, but it will hopefully help the discussion.

Regards,
Christoph

 

[trialotto]

General remark

Somewhat surprised with the contents of your reaction.

It seems that we are not in line when discussing specific topics. Can I suggest to use different topic titles to discuss the various topics?

Hence, the topics would be in my opinion:
- Greylisting behaviour
- Listing Syntax

If you want to add something, please do.

 

[trialotto]

Greylisting Behaviour - Part III

In response to your elaborate posts, I can say the following.

As good as possible, i will respond per item mentioned by you.

First of all, somewhere in my posts i stated:

Since greylisting is a system storing data on triplets, this would be explaining your maillog. Somewhere in the process, your mail system has learned that one of them should be rejected directly.

You responded with something like "no, the maillog explicitly says.... REJECT....due to blacklisting".

This is obvious behaviour, since the greylisting filter has learned from you that the domain should be reject: you instructed the filter by including the domain on a blacklist.

Second, in point 4c you give an example of normal behaviour. At first, the domain is being deferred and after a while, a retry is done by the (legitimate sending) mail server. This time a skip is the result, since that is the essence of greylisting.

However, some issues arise: your mail address [email protected] is certainly a mail address that should not be deferred in the first place, since the domain is on the default white list.

A possible explanation can be the presence of specific DNS setup by yahoo.com. However, i never did receive any defer result on a yahoo message. So, it remains strange.

Third, in response to points 1 to 4b, that is correct.

In my posts i noted that the type of expression does not matter, when using expressions for the white and black list, as long as the IP address can be resolved. This IP address then allows the process of rDNS.

I will post something more about this in the topic Listing Syntax.

 

[Flachzange]

Greylisting behavior part IV

Since greylisting is a system storing data on triplets, this would be explaining your maillog. Somewhere in the process, your mail system has learned that one of them should be rejected directly.

You responded with something like "no, the maillog explicitly says.... REJECT....due to blacklisting".

This is obvious behaviour, since the greylisting filter has learned from you that the domain should be reject: you instructed the filter by including the domain on a blacklist.

ok, I understood the term "learning" as something which does the greylisting by its own. I would expect the greylisting filter to reject something when I have put it on the blacklist, which it certainly does.

Second, in point 4c you give an example of normal behaviour. At first, the domain is being deferred and after a while, a retry is done by the (legitimate sending) mail server. This time a skip is the result, since that is the essence of greylisting.

I gave this example of normal behavior as you have written in some post above, that SKIP is an unusual behavior and I need to reconfigure my mailserver

However, some issues arise: your mail address [email protected] is certainly a mail address that should not be deferred in the first place, since the domain is on the default white list.

A possible explanation can be the presence of specific DNS setup by yahoo.com. However, i never did receive any defer result on a yahoo message. So, it remains strange.

I have cleaned up the above mentioned default white list. This is not the white list used by me. It was just an example.

Thanks for your time and sorry for the misleading topic titles

Christoph

 

[trialotto]

I suppose the behaviour issue can then be closed? If there are any questions, just ask them.

With respect to the syntax issue, I think that there is a profound issue started by you: a probability that the patterns are not being used correctly by the greylisting filter.

Before proceeding with that, can you post your full output from the command grey_listing -i ?

 

[Flachzange]

this is the output from grey_listing -i

Grey listing configuration.

Grey listing checking  enabled
Grey interval          5 minutes
Expire interval        51840 minutes
Penalty interval       2 minutes
Penalty                disabled
Personal grey listing
configuration          prohibited

Server-wide black list:

Server-wide white list:

White domains patterns list:
 fmmailgate*.web.de
 *.rzone.de
 mxpool*.ebay.com
 mxphxpool*.ebay.com
 *.amazon.com
 *.obsmtp.com
 mail*.google.com
 mailout*.t-online.de
 mail.gmx.net
 pmt.perfora.net
 memailout*.messagereach.com
 smtpsrv*.fagms.de
 smtp*.netcologne.de
 mailgate.db-group.com
 mail*.srv2.de
 *.rz.ruhr-uni-bochum.de
 *.paypal.com
 *.rz.RWTH-Aachen.DE
 *.alpha.ec-cluster.com
 mout*.freenet.de
 mta.em.mathworks.com
 smtprelay*.ispgateway.de
 moutng.kundenserver.de
 facebookmail.com
 mx.newstroll.de
 *-out-*.google.com
 *.emirates.com

Black domains patterns list:
 *cam.directnet.com.br
 dynamic|ppp|dyn-ip|dial-up|dsl|dhcp
 *newssystem.info
 *pool.einsundeins.de
 *beta.cmk.ru
 *.ip*.fastwebnet.it
 *[0-9][0-9][0-9].[0-9][0-9][0-9].[0-9][0-9][0-9]*
 *[0-9][0-9]-[0-9][0-9]-[0-9][0-9]*
 *[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9]*
 *[0-9][0-9].[0-9][0-9].[0-9][0-9]*
 *mail.ellink.ru*
 *internetdsl.tpnet.pl*
 [0-9][0-9].[0-9][0-9].[0-9][0-9]*
 [0-9][0-9][0-9].[0-9][0-9][0-9].[0-9][0-9][0-9]*
 *dsl*
 *servidoresdns.net*
 *dynamic.hinet.net*
 *dynamic*
 *[0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*
 mailrelay.daycohost.net
 *.dsl*
 *dsl.*
 *dxap.org
 *cust-adsl.tiscali.it
 ugtel.ru
 *rev.gaoland.net
 *ipcom.comunitel.net
 *dclient.hispeed.ch
 *.altanusprivatemedia.com
 *.retail.telecomitalia.it
 *.camilodossantos.com.br
 *.mrse.com.ar
 *static*.business.telecomitalia.it
 ppp*.iol.it
 newsletter*.web.de
 relay.zaural.ru
 cpt-crt.axnet.com.br
 mail.webm.ru
 ns.krasfin.r
SUCCESS: Gathering of server wide information complete.

 

[trialotto]

Syntax

Flachzange,

It took some time for me to respond and this has to do with lack of answers.

Sure, you have a lot of double entries in your lists, but that does not seem to matter. There is no priority on the basis of the list position.

Furthermore, whatever test i run, i cannot duplicate the error/strange behaviour you were getting (the reason why this thread has been started).

However, in all tests i did run, i also noticed that the overall effectiveness of greylisting is poor (independent of settings and configuration). That is also true for spamdyke, but that filter is at least to some acceptable degree efficient.

In my opinion, greylisting filter being on or off does not really matter, since everything else can be captured by the other spam filters that are present in the standard plesk installation.

The only greylisting filter that should be considered is spamdyke, if even necessary.

 

[Flachzange]

Hi trialotto,

thanks for taking your time again.

Sure, you have a lot of double entries in your lists, but that does not seem to matter. There is no priority on the basis of the list position.

" a lot" is a bit exaggerated There some double entries, which is mainly related to my syntax problem. (testing purpose). But as you said, this doesnt matter.

However, in all tests i did run, i also noticed that the overall effectiveness of greylisting is poor (independent of settings and configuration). That is also true for spamdyke, but that filter is at least to some acceptable degree efficient.

In my opinion, greylisting filter being on or off does not really matter, since everything else can be captured by the other spam filters that are present in the standard plesk installation.

For me greylisting works great (except the issue withe the black and whitelisting) 99,99 percent of all my spam is caught by the greylisting filter and the rest is done by spamassassin. Its advantage is that spam mails are rejected immediately and are not processed by further spam filters like spamassassin. Furthermore, in 99,9999% of the spam mails I can be sure that it really is spam.


But back to my problem. I have an (whitelisting) example which you could check for your own, in case you have access to some facebook.com mails.

Nov 1 20:46:52 h1610138 relaylock: /var/qmail/bin/relaylock: mail from 69.63.178.170:41964 (outmail011.snc1.tfbnw.net)
Nov 1 20:46:53 host qmail-queue-handlers[13899]: Handlers Filter before-queue for qmail started ...
Nov 1 20:46:53 host qmail-queue-handlers[13899]: [email protected]
Nov 1 20:46:53 host qmail-queue-handlers[13899]: [email protected]
Nov 1 20:46:53 host qmail-queue-handlers[13899]: hook_dir = '/opt/psa/handlers/before-queue'
Nov 1 20:46:53 host qmail-queue-handlers[13899]: recipient[3] = '[email protected]'
Nov 1 20:46:53 host qmail-queue-handlers[13899]: handlers dir = '/opt/psa/handlers/before-queue/recipient/[email protected]'
Nov 1 20:46:53 host qmail-queue-handlers[13899]: call_handlers: call executable = '/opt/psa/handlers/info/05-grey-6It4LH/executable'
Nov 1 20:46:53 host greylisting filter[13905]: Starting greylisting filter...
Nov 1 20:46:53 host qmail-queue-handlers[13899]: handlers_stderr: DEFER
Nov 1 20:46:53 host qmail-queue-handlers[13899]: call_handlers: DEFER during call '/opt/psa/handlers/info/05-grey-6It4LH/executable' handler
Nov 1 20:46:53 host qmail-queue-handlers[13899]: call_handlers: stop call handlers from dir '/opt/psa/handlers/before-queue/global'

So as you can see is the mail from facebook not whitelisted. First I only had "tfbnw.net" in my white list, as this the reverse DNS domain, then I tried all the other options (see below). But without success, mails from facebook are still greylisted.

I hope this will help.


Christoph


my current white list:

Server-wide white list:

White domains patterns list:
 fmmailgate*.web.de
 *.rzone.de
 mxpool*.ebay.com
 mxphxpool*.ebay.com
 *.amazon.com
 *.obsmtp.com
 mail*.google.com
 mailout*.t-online.de
 mail.gmx.net
 pmt.perfora.net
 memailout*.messagereach.com
 smtpsrv*.fagms.de
 smtp*.netcologne.de
 mailgate.db-group.com
 mail*.srv2.de
 *.rz.ruhr-uni-bochum.de
 *.paypal.com
 *.rz.RWTH-Aachen.DE
 *.alpha.ec-cluster.com
 mout*.freenet.de
 mta.em.mathworks.com
 smtprelay*.ispgateway.de
 moutng.kundenserver.de
 *facebookmail.com
 mx.newstroll.de
 *-out-*.google.com
 *.emirates.com
 *tfbnw.net
 tfbnw.net
 facebookmail.com

 

[trialotto]

Flachzange,

I quote you for ease of reference:

For me greylisting works great (except the issue withe the black and whitelisting) 99,99 percent of all my spam is caught by the greylisting filter and the rest is done by spamassassin. Its advantage is that spam mails are rejected immediately and are not processed by further spam filters like spamassassin. Furthermore, in 99,9999% of the spam mails I can be sure that it really is spam.

True! A lot of spam is captured, so true. Rejection of spam messages is also an advantage.

But I am talking about other advantages/disadvantages.

In order to keep the discussion alive for others, a minor summary:
PRO: spam messages rejected (not entering the system), if they are really spam and correctly identified as spam
CON: you have to put a lot of effort in tracking true spam addresses and spammers are creative in that
PRO: effective due to explicit whitelisting of domains and mail addresses
CON: black listing not really effective (creativity of spammers!) and time consuming to update
PRO: integration with spamassassin, for whitelisted mail addresses: does relieve spamassassin from some work and makes your system faster
CON: integration with spamassassin, for blacklisted mail addresses: same effectiveness as whitelists when blocking "known" mail addresses, but there we have the creativity again
CON: overall inefficiency, since the mail system is requiring good mails to be send again, if deferred first: the system's efficiency is decreasing
CON: overall inefficiency, since the probability on false negatives (reject or defer when it is not spam) is too large (!)

However, in general, the greylisting filter (as it is now) can be made effective but only due to the integration with spamassassin.

True, it is also desirable that spam is not returning when being deferred. But note that true spammers will try endlessly (i noted that already), since they do anticipate a greylisting filter

A real PRO for the greylisting filter (!!!!!): use of the filter in separate mail server that acts as a filter/relay for the final mail servers.

PS By the way, your facebook issue is likely due to the missing wildcard: try *facebookmail.com