Question Email blacklists

[QWeb Ric]

In Plesk's server-wide mail settings there's a blacklist we can populate, in addition to listing other DNSBL services within the settings tab.

I'm just wondering, is it possible to just populate one single Plesk server's blacklist and then point other Plesk servers to this one as a DNSBL? I manage a bunch of Plesk servers in addition to our own and having to ban the same domains across them all is pretty cumbersome. If I could just set one of our own servers up as a blacklist this would be amazing.

Regards.

 

[mr-wolf]

A google search gave me this: http://www.kloth.net/internet/dnsbl-howto.php

But lucky for you no need to do all that with Plesk.
All you need to do is create a subdomain on your authorative DNS-server with Plesk on your own domain. You can disable the website.

If you have the domain qweb.com
Just create the subdomain: dnsbl.qweb.com

Then create the A-record

2.0.0.127.dnsbl.qweb.com.    IN A   127.0.0.2

And TXT-record

2.0.0.127.dnsbl.qweb.com.    IN TXT   "Blocked"

You can create an SPF-record to tell others that no-one should use the domain for mailing

dnsbl.qweb.com.    IN TXT "v=spf1 a -all"

If you need to blacklist all IP's in the range: 10.222.5.0/24 you need to create an A-record:

*.5.222.10.dnsbl.qweb.com.    IN A   127.0.0.2

And that's it.
You can also create a TXT-record for that domain (*.5.222.10.dnsbl.qweb.com), but if none is there the TXT-record of 2.0.0.127.dnsbl.qweb.com is used.
An explaining website link can be given there, but that's not needed if it is for own use.

host 75.5.222.10.dnsbl.qweb.com
75.5.222.10.dnsbl.qweb.com has address 127.0.0.2

Tested all this and I now have my own dnsbl list..... ;-)
I will think about it if I will start using it.

 

[QWeb Ric]

Thanks, but I'm not sure I follow exactly...

In Plesk -> Tools & Settings -> Mail Server Settings -> Black List, we're currently blocking 1987 domains (in addition to SPF filtering and using other DNSBL services). We block new domains daily and on every Plesk server we look after, we're adding to this black list independently. I.e. these 1987 blocks are replicated on multiple separate servers.

What I'm wanting to do is continue maintaining this list on just the one server, and then add this server as a DNSBL service to the others.

 

[QWeb Ric]

Perhaps DNSBL is the wrong terminology for this, actually. I presumed the black list within Plesk worked as one already and that I just needed a way of pointing other Plesk servers to it as a public service, but thinking about it further, this may not make as much sense as I originally thought, because two domains might point to the same IP address and as far as I'm aware, blocking a domain via this list doesn't actually block that domain's IP address, it blocks literally just the one domain. It must therefore work as a filter against the email headers rather than the originating server address... I think...

I do hope this is possible =).

 

[mr-wolf]

It is possible to do this, exactly as I explained. I have no idea why you dismiss it.
It's done in 5 minutes.

You may not know that you need DNSBL, but it's created exactly for this purpose. Whether you use it privately or for others is irrelevant

For IP's it's DNSBL, for domains it's URIBL.
It works the same.
Just add the domain name and point it to 127.0.0.2


I would use another subdomain for it.

uribl.qweb.com


Plesk supports DNSBL out of the box. You need to add support for URIBL to postfix yourself. Because I use an anti-spam reverse proxy (ASSP) I have never done that, but it's presumably a single line in /etc/postfix/main.cf.



Edit:

I looked it up for you.
You only need to add your uribl to an existing line of your postfix.

Add ", reject_rbl_client uribl.qweb.com" to that line.

grep smtpd_recipient_restrictions /etc/postfix/main.cf

smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client uribl.qweb.com

 

[QWeb Ric]

Thanks. I'm not dismissing your answer, I'm just not quite sure that I understand it.

If I amend the Postfix config as per your answer, and set up the DNS records as per your earlier answer, then a) would we need to replace all of the blocks currently set up in Plesk -> Tools & Settings -> Mail Server Settings -> Black List with DNS records? And b) would it literally just be a case then of adding our uribl subdomain to the DNSBL list within the Plesk settings on the other servers that we manage so that they use it?

If this is the way to do it then it's going to be a headache to replace all of these entries, but on the other hand our nameservers permit API access so we could potentially add/remove DNS records programmatically, which would be pretty handy.

 

[mr-wolf]

Yes....
But it's the dnsbl subdomain, not uribl subdomain. (dnsbl.qweb.com)

And if you have those entries in a file I can script that including the translation of cidr notation.
That translation would only be able to do plain IP's and in CIDR-notation the /32, /24, /16 and /8 blocks.

All other CIDR's will be ignored (/23, /22, /20, /19....)
It's more complex to write a script supporting those, but not impossible
A /23 would become 2 /24 addresses, but this is already nice.
It generates a list of the CIDR's that are ignored...
You could add these manually

EDIT 30-07-2017:
I streamlined the code somewhat.

cat /usr/local/sbin/dnsbl

# cat /usr/local/sbin/dnsbl
#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.qweb.com
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

# Filter IPv4 addresses, 1 per row, only spaces or tabs allowed in front
egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t'| sort -uo ${TMPDIR}/all_blocks
echo "Found `grep -c '.' ${TMPDIR}/all_blocks` IP-blocks " | tee -a ${TMPLOG}

# Filter the plain IP's and CIDR with /32,/24,/16 and /8
egrep -o '^[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/32|[0-9]+/24|[0-9]+/16|[0-9]+/8)$' ${TMPDIR}/all_blocks >${TMPDIR}/blocks

echo "I was able to use `grep -c '.' ${TMPDIR}/blocks` IP-blocks " | tee -a ${TMPLOG}
echo -e "These CIDR's are ignored:\n`grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks`\n\n"  | tee -a ${TMPLOG}

while read BLOCK ; do
  # remove the CIDR, Split the IPv4 in 4 segments keeping only the digits and reverse the order (tac)
  echo "${BLOCK}" | sed 's/\/.*//g' | egrep -o '[0-9]+($|\.)' | tr -d '.' | tac >${TMPDIR}/ip4

  ASTERISKS=0                                         # /32 or plain
  echo "${BLOCK}" | grep -q "/24" && ASTERISKS=1      # /24
  echo "${BLOCK}" | grep -q "/16" && ASTERISKS=2      # /16
  echo "${BLOCK}" | grep -q "/8"  && ASTERISKS=3      # /8

  N=1
  REV_IP=""
  while read IP4 ; do
    if [ $N -ge ${ASTERISKS} ] ; then
      if [ $N -eq ${ASTERISKS} ] ; then
        REV_IP="${REV_IP}*."            # Replace whatever is there with an asterisk
      else
        REV_IP="${REV_IP}${IP4}."       # Write the IPv4 segment
      fi
    fi
    let N+=1
  done<${TMPDIR}/ip4
  # Write the reversed IP to the file after stripping the superfluous last dot
  echo "${REV_IP}" | sed 's/\.$//g' >>${TMPDIR}/rev_ip4

done<${TMPDIR}/blocks

if [ -s ${TMPDIR}/rev_ip4 ] ; then
  # Add the IPBLOCKS to DNS
  N=1
  while read IPBLOCK ; do
    echo "${N}. Add ${IPBLOCK}.${DNSBL} IN A ${LOCALHOST}" | tee -a ${TMPLOG}
    ${PLESKDNS} --add ${DNSBL} -a "${IPBLOCK}" -ip ${LOCALHOST} | tee -a ${TMPLOG}
    let N+=1
  done<${TMPDIR}/rev_ip4

  cat ${TMPLOG} >>${LOG}        # Write temporary log to log
fi

rm -r ${TMPDIR}

 

[mr-wolf]

You kept me busy this weekend!!
I couldn't stop thinking about those remaining subnets that I had to skip because they would not fit into a wildcard.
So I thought it over some more and decided to write something that would break down those non-processed subnets into processable subnets (/8, /16, /24 & /32)

It does need the binary ipcalc (apt install ipcalc)

If the program "aggregate" is installed (it is mandatory for the complementary script) it will be used with the option "-t". This option gives a warning when it's fed a mismatched subnet and will convert it into a subnet where the prefix is leading.

This is the result:

cat /usr/local/sbin/dnsbl

#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.qweb.com
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2
BINDDIR=/var/named/run-root/var
DNSFILE=${BINDDIR}/${DNSBL}

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

# The program will work better with aggregate, it uses 'cat' otherwise
# With aggregate it will correct netblocks with mismatched suffix (like 192.168.10.20/24 => 192.168.10.0/24)
AGGREGATE="cat"
if which aggregate >/dev/null 2>&1 ; then
  AGGREGATE="aggregate -t"
fi

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

if [ ! -d "${BINDDIR}" ] ; then
  echo "The folder ${BINDDIR} does not exist, quitting"
  exit 1
fi

if [ ! -f "${DNSFILE}" ] ; then
  echo "The DNS-file ${DNSFILE} does not exist (is this an authorative DNS-server?), quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

# Filter IPv4 addresses, 1 per row, only spaces or tabs allowed in front
egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t'| sort -uo ${TMPDIR}/all_blocks
echo "Found `grep -c '.' ${TMPDIR}/all_blocks` IP-blocks " | tee -a ${TMPLOG}

# Filter the plain IP's and CIDR with /32,/24,/16 and /8
egrep -o '^[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/32|[0-9]+/24|[0-9]+/16|[0-9]+/8)$' ${TMPDIR}/all_blocks >${TMPDIR}/blocks

grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks | ${AGGREGATE} >${TMPDIR}/rest_blocks

if ! which ipcalc >/dev/null ; then
  echo "I was able to use `grep -c '.' ${TMPDIR}/blocks` IP-blocks " | tee -a ${TMPLOG}
  echo -e "These CIDR's are ignored:\n`grep -vf ${TMPDIR}/blocks ${TMPDIR}/all_blocks`\n\n"  | tee -a ${TMPLOG}
else
  echo "I have `grep -c . ${TMPDIR}/rest_blocks` remaining blocks that I need to break down"
  while read BLOCK ; do
    NETWORK=`ipcalc -n "${BLOCK}" | awk -F= '{print $2}'`
    BROADCAST=`ipcalc -b "${BLOCK}" | awk -F= '{print $2}'`
    SUFFIX=`echo "${BLOCK}" | sed 's/.*\///g'`

    BLOCKS=`ipcalc -m "${NETWORK}/${SUFFIX}" | egrep -o '[0-9]+' | grep -c '255'`
    echo "${NETWORK}" | egrep -o '[0-9]+' >${TMPDIR}/segments

    N=1
    BASE=""
    echo -n '' >${TMPDIR}/brokendownblocks
    while read SEGMENT ; do
      [ ${BLOCKS} -lt ${N} ] && break
      BASE="${BASE}${SEGMENT}."
      let N+=1
    done<${TMPDIR}/segments

    START="`echo ${NETWORK} | sed "s/${BASE}//g" | sed 's/\..*//g'`"
    END="`echo ${BROADCAST} | sed "s/${BASE}//g" | sed 's/\..*//g'`"

    NEW_SUFFIX="/32"
    if [ ${SUFFIX} -lt 16 ] ; then
      if [ ${SUFFIX} -lt 8 ] ; then
        NEW_SUFFIX=".0.0.0/8"
      else
        NEW_SUFFIX=".0.0/16"
      fi
    elif [ ${SUFFIX} -lt 24 ] ; then
      NEW_SUFFIX=".0/24"
    fi

    N=${START}
    while [ ${N} -le ${END} ]  ; do
      echo "${BASE}${N}${NEW_SUFFIX}" >>${TMPDIR}/brokendownblocks
      let N+=1
    done
    echo "I broke down ${NETWORK}/${SUFFIX} into `grep -c . ${TMPDIR}/brokendownblocks` ${NEW_SUFFIX} blocks"
    cat ${TMPDIR}/brokendownblocks >>${TMPDIR}/blocks
  done<${TMPDIR}/rest_blocks
  echo "I now have in total `grep -c . ${TMPDIR}/blocks` blocks"
fi

I=1
while read BLOCK ; do
  echo -en "${I}     \r"

  # remove the CIDR, Split the IPv4 in 4 segments keeping only the digits and reverse the order (tac)
  echo "${BLOCK}" | sed 's/\/.*//g' | egrep -o '[0-9]+($|\.)' | tr -d '.' | tac >${TMPDIR}/ip4

  ASTERISKS=0                                         # /32 or plain
  echo "${BLOCK}" | grep -q "/24" && ASTERISKS=1      # /24
  echo "${BLOCK}" | grep -q "/16" && ASTERISKS=2      # /16
  echo "${BLOCK}" | grep -q "/8"  && ASTERISKS=3      # /8

  N=1
  REV_IP=""
  while read IP4 ; do
    if [ $N -ge ${ASTERISKS} ] ; then
      if [ $N -eq ${ASTERISKS} ] ; then
        REV_IP="${REV_IP}*."            # Replace whatever is there with an asterisk
      else
        REV_IP="${REV_IP}${IP4}."       # Write the IPv4 segment
      fi
    fi
    let N+=1
  done<${TMPDIR}/ip4
  # Write the reversed IP to the file after stripping the superfluous last dot
  echo "${REV_IP}" | sed 's/\.$//g' >>${TMPDIR}/rev_ip4
  let I+=1
done<${TMPDIR}/blocks

if [ -s ${TMPDIR}/rev_ip4 ] ; then
  # Add the IPBLOCKS to DNS
  N=1
  while read IPBLOCK ; do
    if grep -q "^${IPBLOCK}.${DNSBL}.*IN.*A" ${DNSFILE} ; then
      echo "${N}. ${IPBLOCK}.${DNSBL} already added"
    else
      echo "${N}. Add ${IPBLOCK}.${DNSBL} IN A ${LOCALHOST}" | tee -a ${TMPLOG}
      ${PLESKDNS} --add ${DNSBL} -a "${IPBLOCK}" -ip ${LOCALHOST} | tee -a ${TMPLOG}
    fi
    let N+=1
  done<${TMPDIR}/rev_ip4

  cat ${TMPLOG} >>${LOG}        # Write temporary log to log
fi

rm -r ${TMPDIR}

 

[mr-wolf]

As I can now create DNS-entries on my Plesk server using a file containing subnets I also would like to be able to do the reverse.
That's why I wrote this script.
It will gather all the IP's (in reverse format) from the DNS-file in /var/named/run-root/var/ and aggregate them into efficient subnets.

This complements the other script

I am getting the same IP' subnets that I had originally, so that's almost proof that the program works correct. It's at least proof that it's not making a mess of it.

cat /usr/local/sbin/dns2bl

#!/bin/bash

BLOCKLIST=/opt/ASSP/files/blockip.txt
DNSBL=dnsbl.qweb.com
PLESKDNS=/usr/local/psa/bin/dns
LOCALHOST=127.0.0.2
BINDDIR=/var/named/run-root/var

DNSFILE=${BINDDIR}/${DNSBL}

THISSCRIPT="`readlink -f $0`"
SCRIPTNAME=${THISSCRIPT##*/}
BASE=${THISSCRIPT##*/}
[ -z "${BASE}" ] && BASE=${0##*/}
LOG=/var/log/${BASE}.log

if [ ! -e "${BLOCKLIST}" ] ; then
  echo "The blocklist ${BLOCKLIST} does not exist, quitting"
  exit 1
fi

TMPDIR=`mktemp -t -d ${0//*\/}.XXXXXXXXXX`
TMPLOG=${TMPDIR}/log
date >${TMPLOG}

egrep -o '^\s*[0-9]+\.[0-9]+\.[0-9]+\.([0-9]+|[0-9]+/[0-9]+)(\s|$)' "${BLOCKLIST}" | tr -d ' \t' | sed -r 's/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/&\/32/g' | aggregate -t 2>/dev/null > ${TMPDIR}/blocklist

egrep "^(\*|[0-9]+)\.[0-9.]+${DNSBL}.*IN.*A.*127" ${DNSFILE} | sed "s/\.${DNSBL}.*//g;s/^\*\.//g" | grep -v "\.0\.0\.127" >${TMPDIR}/allips

if [ ! -s ${TMPDIR}/allips ] ; then
  echo "No DNS-entries found in Zone ${DNSBL}"
else
  touch ${TMPDIR}/reversed_raw
  while read IP ; do
    # Create a reversed list of the IP
    echo "${IP}" | egrep -o '[0-9]+' | tac >${TMPDIR}/segments

    N=1
    REVERSED_IP=""
    while read SEGMENT ; do
      if [ $N -eq 1 ] ; then
        REVERSED_IP="${SEGMENT}"
      else
        REVERSED_IP="${REVERSED_IP}.${SEGMENT}"
      fi
      let N+=1
    done<${TMPDIR}/segments

    if   [ $N -eq 5 ] ; then
      REVERSED_IP="${REVERSED_IP}/32"
    elif [ $N -eq 4 ] ; then
      REVERSED_IP="${REVERSED_IP}.0/24"
    elif [ $N -eq 3 ] ; then
       REVERSED_IP="${REVERSED_IP}.0.0/16"
    else
      REVERSED_IP="${REVERSED_IP}.0.0.0/8"
    fi
    echo "${REVERSED_IP}" >>${TMPDIR}/reversed_raw
  done<${TMPDIR}/allips
  cat ${TMPDIR}/reversed_raw  | aggregate -t 2>/dev/null >${TMPDIR}/reversed

  echo -e "Subnets gathered from DNS:\n"  >&2
  cat ${TMPDIR}/reversed
  echo -e "\n\nSubnets in ${BLOCKLIST}:\n`cat ${TMPDIR}/blocklist`\n\n" >&2
  echo -e "Difference:\n`diff ${TMPDIR}/blocklist ${TMPDIR}/reversed`\n\n" >&2
fi

rm -r ${TMPDIR}

 

[QWeb Ric]

Thanks, this is all massively appreciated.

Now that I've had chance to look over all of the above, am I right in thinking that:

- your scripts basically convert a file of domains in to their corrosponding IP's to then be added to a DNSBL as per your original replies. So for example google.com would convert to 216.58.206.78 so your script would determine that the entry to add to the DNSBL would then be 78.206.58.216.dnsbl.qweb.co.uk

- I'd then add this entry as an A record with a corrosponding TXT record, to the qweb.co.uk DNS.

- Provided dnsbl.qweb.co.uk is then added to Plesk as a blacklist service, email from @google.com would then be blocked because the above A record exists.

If this is correct then I think I follow everything you're doing here =).

My only concern with this approach though, is that where multiple domains share a single IP address, blocking one of those domains will effectively block them all. I suppose that's where URIBL has the advantage but then I'd need to modify the Postfix configurations manually rather than being able to use built-in Plesk functionality? I take it with that approach, there's be the risk of Plesk overwriting the amended configs on occasion too.

 

[mr-wolf]

Thanks, this is all massively appreciated.

Now that I've had chance to look over all of the above, am I right in thinking that:

- your scripts basically convert a file of domains in to their corrosponding IP's to then be added to a DNSBL as per your original replies. So for example google.com would convert to 216.58.206.78 so your script would determine that the entry to add to the DNSBL would then be 78.206.58.216.dnsbl.qweb.co.uk

Exactly.
And if you want to block 216.58.206.0/24 the record *.206.58.216.dnsbl.qweb.co.uk will be created.
Now it may become clear why the IP is in reverse order.
It makes it possible to "abuse" the wildcard mechanism for whole subnets.

You can fill it with /32, /24, /16 and /8 addresses.
If it finds other subnets in your file they will be represented by multiple entries in DNS

- I'd then add this entry as an A record with a corresponding TXT record, to the qweb.co.uk DNS.

No, you don't need a corresponding TXT-record for each entry. Only each response (127.0.0.2 for instance) could do with a TXT-record. That's why you will not see any TXT-records created by my script. Only A-records.

- Provided dnsbl.qweb.co.uk is then added to Plesk as a blacklist service, email from @google.com would then be blocked because the above A record exists.

Exactly.. If it doesn't get an answer, it's good.

If this is correct then I think I follow everything you're doing here =).

My only concern with this approach though, is that where multiple domains share a single IP address, blocking one of those domains will effectively block them all.

More often it's a server you want to block. Often complete subnets.

I suppose that's where URIBL has the advantage but then I'd need to modify the Postfix configurations manually rather than being able to use built-in Plesk functionality?

Yes, that's where the URIBL comes in.
BTW... this can also be used for whitelisting (IP's and URI's)

I take it with that approach, there's be the risk of Plesk overwriting the amended configs on occasion too.

You need to monitor that anyhow.
I have a script that runs every hour and checks several folders. If it detects a change it will tar the folder to a file with a datestamp. This way you can always detect a change.
It's not the end of the world if it loses this.

Plesk doesn't rewrite the config as often as it did and it will now leave some modifications as is. Because it's an added line instead of a changed line it has a better chance for surviving.
I think it will stick...

[EDIT] It's not an added line, but a changed line... (it's been some weeks now). But like I said.... Plesk is not as aggressive as it used to be.

 

[QWeb Ric]

OK thanks, that all makes sense.