Question Integrating AbuseIPDB RealTime IP Check, possibly using ModSecurity and LUA

[Ehud]

Server operating system version

Linux/Ubuntu

Plesk version and microupdate number

Plesk Obsidian

Hi,


I would like to check IPs connecting ports as 80 and 443, on real-time, against data base of abusing IPs, as abuseupdb.com.

I have an API key, which allows curl checks. A test may look like this:

curl -G https://api.abuseipdb.com/api/v2/check \
--data-urlencode "ipAddress=118.25.6.39" \
-d maxAgeInDays=90 \
-d verbose \
-H "Key: my_key" \
-H "Accept: application/json"

Where the results would look like this:

{"data":{"ipAddress":"118.25.6.39","isPublic":true,"ipVersion":4,"isWhitelisted":false,"abuseConfidenceScore":0,"countryCode":"CN","usageType":"Data Center\/Web Hosting\/Transit","isp":"Tencent Cloud Computing (Beijing) Co. Ltd","domain":"tencent.com","hostnames":[],"countryName":"China","totalReports":0,"numDistinctUsers":0,"lastReportedAt":"2022-03-29T11:50:40+00:00","reports":[]}}

I also use ModSecurirty, that enables LUA integration. This is done via a call of @inspectFile, which may call for Pearl of LUA files.

SecRule FILES_TMPNAMES "@inspectFile /etc/apache2/modsecurity.d/modsec_clamav.pl" \
"id:'99999932471', \
phase:2, \
t:none, \
deny, \
log, \
msg:'Infected File upload detected', \
tag:'MALICIOUS_SOFTWARE/VIRUS'"

Note: I'm currently not asking about a check a 10,000 IPs downloaded list, but of a real time check. A check against a downloaded partial list should be possible via CSF.

May I ask if you would know exactly how to create a real-time, one-time-per IP's session, check of remore_addr IP connecting to the server (ports 80/443?) against such abuseipdb, including setting comfort level for the IP being abusive and fetching data from curl JSON results?

 

[MarkM]

If you're using CSF, you can setup a bash script that will tail your logs, check against their db, and then csf deny it if needed.

Something like this essentially;

GitHub - ShaneOss/AbuseIPDB-cPanel-CSF: Script to parse the IPs in the cPanel Apache error log. Checks them against AbuseIPDB and add to CSF deny list if certain criteria is met.

Script to parse the IPs in the cPanel Apache error log. Checks them against AbuseIPDB and add to CSF deny list if certain criteria is met. - GitHub - ShaneOss/AbuseIPDB-cPanel-CSF: Script to parse...

github.com

 

[Ehud]

Hi Mark,

Thanks!

I became aware of that option, however Plesk Support recommendation is not to use CSF in parallel to the Plesk Fire Wall.

 

[Ehud]

Would you say, such script can be called by ModSecurity using the @inspectFile directive?

AbuseIPDB-cPanel-CSF/checkipabuseipdb.sh at master · ShaneOss/AbuseIPDB-cPanel-CSF

Script to parse the IPs in the cPanel Apache error log. Checks them against AbuseIPDB and add to CSF deny list if certain criteria is met. - AbuseIPDB-cPanel-CSF/checkipabuseipdb.sh at master · Sh...

github.com

 

[Ehud]

Hi,

The below is the checking part of the referred script. I'll start checking if and how may I use it.

May I ask if you would have any thoughts regarding adjustment to the below code, that will make it suitable for running via a ModSecurity rule?

#!/bin/bash

# Your own AbuseIPDB API Key
abuseipdbAPIKey="------------------------------- YOUR API KEY -----------------------------------"


# If IP isn't already blocked.
# Check IP address against AbuseIPDB.
read abuseipdbtotalReports abuseipdbconfidenceScore abuseipdbisPublic abuseipdbisWhitelisted abuseipdbcountryName abuseipdblastReportedAt abuseipdbisp abuseipdbusageType abuseipdomain < <(echo $(curl --silent -G https://api.abuseipdb.com/api/v2/check \
--data-urlencode "ipAddress=$ip" \
-d maxAgeInDays=90 \
-d verbose \
-H "Key: $abuseipdbAPIKey" \
-H "Accept: application/json" --stderr - \
| jq -r '.data.totalReports,.data.abuseConfidenceScore,.data.isPublic,.data.isWhitelisted,.data.countryName,.data.lastReportedAt,.data.isp,.data.usageType,.data.domain'))

#echo "totalReports:" $abuseipdbtotalReports "abuseConfidenceScore:" $abuseipdbconfidenceScore "isPublic:" $abuseipdbisPublic "isWhitelisted:" $abuseipdbisWhitelisted "countryName:" $abuseipdbcountryName "lastReportedAt:" $abuseipdblastReportedAt "isp:" $abuseipdbisp "usageType:" $abuseipdbusageType "domain:" $abuseipdomain

# If IP address is found at AbuseIPDB and abuseConfidenceScore > 25 && isPublic:true && isWhitelisted:false
if [[ $abuseipdbconfidenceScore -gt 25 && $abuseipdbisPublic -eq "true" && $abuseipdbisWhitelisted -eq "false" ]];
then
# Block IP address via ModSecurity by returning false
return false
elif [ "$abuseipdbtotalReports" == "null" ];
then
echo "AbuseIPDB API quota check limit may be exceeded. Please try again in 24hrs.";
break;
elif [ $abuseipdbtotalReports -eq 0 ];
then
echo $ip "IP not recorded at AbuseIPDB.";
elif [ "$abuseipdbWhitelisted" = "true" ];
then
echo $ip "IP address whitelisted at AbuseIPDB.";
elif [ "$abuseipdbisPublic" = "false" ];
then
echo $ip "IP is within a private address range and not public.";
elif [[ $abuseipdbconfidenceScore -le 25 ]];
then
echo $ip "AbuseIPDB abuse confidence score < 25%).";
else
echo "Condition not matched.";
echo "totalReports:" $abuseipdbtotalReports "abuseConfidenceScore:" $abuseipdbconfidenceScore "isPublic:" $abuseipdbisPublic "isWhitelisted:" $abuseipdbisWhitelisted "countryName:" $abuseipdbcountryName "lastReportedAt:" $abuseipdblastReportedAt "isp:" $abuseipdbisp "usageType:" $abuseipdbusageType "domain:" $abuseipdomain;
fi
fi
done

 

[Ehud]

Hi,

For the last couples of days I'm struggling with having ModSecurity execute a bash script. This script dynamically checks an IP against abuseIPDB. The script works if activated as a stand alone, however fails, when called from a ModSecurity rule.

When run on its own I insert a specific IP to be tested, and it works, while when tested combined with ModSecurity I use a VAR which is exported from ModSecurity using 'setenv'.

The Rule I use is:

# Rule - Currently not in place
SecRule REMOTE_ADDR "!@ipMatch 127.0.0.1,my-ip" \
"id:'99999932480', \
phase:2, \
t:none, \
t:lowercase, \
deny, \
logdata:'{ip=%{REMOTE_ADDR}}', \
log, \
auditlog, \
setenv:remote_addr=%{REMOTE_ADDR}, \
msg:'AbuseIPDB detected the IP as Malicious', \
tag:'MALICIOUS_SOFTWARE/VIRUS', \
exec:/etc/apache2/modsecurity.d/checkAbuseipdb.sh"

The script I use is:

/etc/apache2/modsecurity.d/checkAbuseipdb.sh

#!/bin/bash
#Check IP address against AbuseIPDB.
read abuseipdbtotalReports abuseipdbconfidenceScore abuseipdbisPublic abuseipdbisWhitelisted abuseipdbcountryName abuseipdblastReportedAt abuseipdbisp abuseipdbusageType abuseipdomain < <(echo $(curl --silent -G https://api.abuseipdb.com/api/v2/check \
--data-urlencode "ipAddress=$remote_addr" \
-d maxAgeInDays=90 \
-d verbose \
-H "Key: here-comes-key" \
-H "Accept: application/json" --stderr - \
| jq -r '.data.totalReports,.data.abuseConfidenceScore,.data.isPublic,.data.isWhitelisted,.data.countryName,.data.lastReportedAt,.data.isp,.data.usageType,.data.domain'))
#echo "totalReports:" $abuseipdbtotalReports "abuseConfidenceScore:" $abuseipdbconfidenceScore "isPublic:" $abuseipdbisPublic "isWhitelisted:" $abuseipdbisWhitelisted "countryName:" $abuseipdbcountryName "lastReportedAt:" $abuseipdblastReportedAt "isp:" $abuseipdbisp "usageType:" $abuseipdbusageType "domain:" $abuseipdomain
output="1 Unable to parse abuseIPDB output"
#If IP address is found at AbuseIPDB and abuseConfidenceScore > 25 && isPublic:true && isWhitelisted:false
if [[ $abuseipdbconfidenceScore -gt 25 && $abuseipdbisPublic -eq "true" && $abuseipdbisWhitelisted -eq "false" ]]
then
#Block IP address via ModSecurity by returning false
output="0 AbuseIPDB detected the IP as Malicious."
printf "$output\n"
elif [ "$abuseipdbtotalReports" == "null" ]
then
output="1 AbuseIPDB API quota check limit may be exceeded. Please try again in 24hrs."
printf "$output\n"
elif [ $abuseipdbtotalReports -eq 0 ]
then
output="1 IP not recorded at AbuseIPDB."
printf "$output\n"
elif [ "$abuseipdbWhitelisted" = "true" ]
then
output="1 IP address whitelisted at AbuseIPDB."
printf "$output\n"
elif [ "$abuseipdbisPublic" = "false" ]
then
output="1 IP is within a private address range and not public."
printf "$output\n"
elif [[ $abuseipdbconfidenceScore -le 25 ]]
then
output="1 IAbuseIPDB abuse confidence score < 25%."
printf "$output\n"
else
echo "Condition not matched."
echo "totalReports:" $abuseipdbtotalReports "abuseConfidenceScore:" $abuseipdbconfidenceScore "isPublic:" $abuseipdbisPublic "isWhitelisted:" $abuseipdbisWhitelisted "countryName:" $abuseipdbcountryName "lastReportedAt:" $abuseipdblastReportedAt "isp:" $abuseipdbisp "usageType:" $abuseipdbusageType "domain:" $abuseipdomain
output="1 Unable to parse abuseIPDB output"
printf "$output\n"
fi

****************************************************************************************

This script gets permissions to run:

# Setting Permissions
chmod 750 /etc/apache2/modsecurity.d/checkAbuseipdb.sh
sudo chmod +x /etc/apache2/modsecurity.d/checkAbuseipdb.sh

****************************************************************************************



To test the rule, I use curl with an IP that it found on the the abuseIPDB:

# curl that emulates the real IP as-if connecting the server.
curl -I --header "X-Forwarded-For: tested-ip-here" "Example Domain" --referer "Google" --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.2883.87 Safari/537.36 eM Client/8.2.1659.0"

Error log:

2022-05-05 03:56:28 Error tested-ip-comes-here 403 GET / HTTP/1.0 Google Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.2883.87 Safari/537.36 eM Client/8.2.1659.0 6.04 K Apache SSL/TLS access
2022-05-05 03:56:28 Error tested-ip-comes-here [client tested-ip-comes-here] ModSecurity: Exec: Execution failed while reading output: /etc/apache2/modsecurity.d/checkAbuseipdb.sh (End of file found) [hostname "example.com"] [uri "/"] [unique_id "YnMgvAOWCvLoWp3VchdfMwAAAAA"], referer: Google Apache error
2022-05-05 03:56:28 Error tested-ip-comes-here [client tested-ip-comes-here] ModSecurity: Failed to execute: /etc/apache2/modsecurity.d/checkAbuseipdb.sh [hostname "example.com"] [uri "/"] [unique_id "YnMgvAOWCvLoWp3VchdfMwAAAAA"], referer: Google Apache error
2022-05-05 03:56:28 Error tested-ip-comes-here [client tested-ip-comes-here] ModSecurity: Access denied with code 403 (phase 2). Match of "ipMatch 127.0.0.1,212.xxx.xxx.xxx" against "REMOTE_ADDR" required. [file "/etc/apache2/plesk.conf.d/modsecurity.conf"] [line "292"] [id "99999932480"] [msg "AbuseIPDB detected the IP as Malicious"] [data "{ip=tested-ip-comes-here}"] [tag "MALICIOUS_SOFTWARE/VIRUS"] [hostname "example.com"] [uri "/"] [unique_id "YnMgvAOWCvLoWp3VchdfMwAAAAA"], referer: Google Apache error

****************************************************************************************


If I place that IP instead of the VAR $remote_addr in the script, and run CLI:

# CLI
./checkAbuseipdb.sh

The rule well functions.


****************************************************************************************

Each time I use the above curl, any IP is blocked due to the fact it's not part of the first filter for being not my IP. The script itself does not run...

I also can see on the COUNT abuseIPDB API shows, the count did not change for ModSecurity event, but does for direct running of the script.

****************************************************************************************

On server configuration, I see on the section "Configure Command", the parameter: "without-curl".

****************************************************************************************

exec() is explained on:
Reference Manual (v2.x) · SpiderLabs/ModSecurity Wiki

setenv is explained on:
Reference Manual (v2.x) · SpiderLabs/ModSecurity Wiki

****************************************************************************************



May I ask, if there is any special configuration of Plesk required for such ModSecurity rule to work with the script, possibly it being:

1) curl support on ModSecurity or on Bash scripts

2) setenv support on ModSecurity and afterwards fetching the var outside as intended

3) exec() command being used on ModSecurity

4) Script to be used for ModSecurity to have a top line including 'curl' (although it does work on direct script activation)

 

[weltonw]

Would not recommend this. Run it async with the request as @Mark suggested. Otherwise you'll be potentially experience significant server delays as modsec inspection is blocking.

Don't need to use CSF. Iptables or fail2ban works

 

[Ehud]

Would not recommend this. Run it async with the request as @Mark suggested. Otherwise you'll be potentially experience significant server delays as modsec inspection is blocking.

Don't need to use CSF. Iptables or fail2ban works

Hi John,

Thanks. I understand this aspect, and intended to observe it, or to, some how, serve an Ok, and test the IPs off-line a few seconds later, when still relevant to their activity.

However, due to the fact most attacks rotate their IPs, and rarely use them again on the same target, a post mortem block, would not be, IMHO, that helpful.

 

[mow]

However, due to the fact most attacks rotate their IPs, and rarely use them again on the same target, a post mortem block, would not be, IMHO, that helpful.

In that case, even blocking them directly would not be very helpful either.
Running this for every request can even increase your susceptibility to dDOS.

 

[Ehud]

In that case, even blocking them directly would not be very helpful either.
Running this for every request can even increase your susceptibility to dDOS.

I would adjust, so the checking of an IP would be done only once, within a timeframe. If IP is Ok, server is good. And if IP is blocked.. well, it would be blocked by the FireWall, as of its next connection...

And not that this is the only server protection...

However, out of the 0.1% attacks reaching server, this desired filter could reduce additional 90% of them.

May I ask, if you would have any thoughts regarding ways to make this almost real-time dynamic IP filtering work?

 

[Ehud]

Eureka!



It now works as a charm. The last modification needed, was the change of the script permissions to be 755:

chmod 755 /etc/apache2/modsecurity.d/checkAbuseipdb.sh

sudo chmod +x /etc/apache2/modsecurity.d/checkAbuseipdb.sh

The ModSecurity rule, which currently works, fast as a lightening.

Due to the broad nature of the abuseipdb, this SecRule is one of the strongest and thin cyber security filters I have seen, most likely eliminating an addiotnal 90-95% of the very small sliver the attacks that do reach the server constitute.

 

[Ehud]

SecRule needs some modification.

 

[Ehud]

I have managed to walk-around a problem, where I could not make the BASH script return a STDOUT value to the SecRule.

I did so, by first writing BAD IPs to a log file, and then, testing in a second rule if an IP is on the file.

I have also created SecRule generating sessionID. I still haven't managed to force the check to be once per session.

# Initialise session variables using the session cookie value
SecRule REQUEST_COOKIESHPSESSID !^$ "nolog,pass,id:99999932481,setsid:%{REQUEST_COOKIES.PHPSESSID}"

# This rule will identify the outbound Set-Cookie SessionID data and capture it in a setsid #
SecRule RESPONSE_HEADERS:/Set-Cookie2?/ "(?ij?sessionid|(php)?sessid|(asp|jserv|jw)?session[-_]?(id)?|cf(id|token)|sid)=([^\s]+)\;\s?)" "chain,id:99999932482,phase:3,t:none,pass,nolog,capture,setsid:%{TX.6},setvar:session.sessionid=%{TX.6},setvar:session.valid=1"
SecRule SESSION:SESSIONID "(.*)" "t:none,t:sha1,t:hexEncode,capture,setvar:session.csrf_token=%{TX.1}"

# Log bad IPs to file
SecAction \
"id:'99999932480', \
phase:1, \
t:none, \
t:lowercase, \
pass, \
logdata:'{ip=%{REMOTE_ADDR}}', \
nolog, \
setenv:remote_addr=%{REMOTE_ADDR}, \
msg:'Log to the file /var/log/log.log file IP that AbuseIPDB detected as Malicious', \
exec:/etc/apache2/modsecurity.d/checkAbuseipdb.sh"

# Test if new IP is on file
SecRule REMOTE_ADDR "@ipMatchFromFile /var/log/log.log" \
"id:99999932483, \
phase:2, \
t:none, \
t:lowercase, \
deny, \
log, \
auditlog, \
msg:'AbuseIPDB detected the IP as Malicious', \
tag:'MALICIOUS_SOFTWARE/VIRUS'"

New script is:

#!/bin/bash

#Check IP address against AbuseIPDB.
read abuseipdbtotalReports abuseipdbconfidenceScore abuseipdbisPublic abuseipdbisWhitelisted abuseipdbcountryName abuseipdblastReportedAt abuseipdbisp abuseipdbusageType abuseipdomain < <(echo $(curl --silent -G https://api.abuseipdb.com/api/v2/check \
--data-urlencode "ipAddress=$remote_addr" \
-d maxAgeInDays=90 \
-d verbose \
-H "Key: your_key" \
-H "Accept: application/json" --stderr - \
| jq -r '.data.totalReports,.data.abuseConfidenceScore,.data.isPublic,.data.isWhitelisted,.data.countryName,.data.lastReportedAt,.data.isp,.data.usageType,.data.domain'))

#echo "totalReports:" $abuseipdbtotalReports "abuseConfidenceScore:" $abuseipdbconfidenceScore "isPublic:" $abuseipdbisPublic "isWhitelisted:" $abuseipdbisWhitelisted "countryName:" $abuseipdbcountryName "lastReportedAt:" $abuseipdblastReportedAt "isp:" $abuseipdbisp "usageType:" $abuseipdbusageType "domain:" $abuseipdomain

#If IP address is found at AbuseIPDB and abuseConfidenceScore > 25 && isPublic:true && isWhitelisted:false
if [[ $abuseipdbconfidenceScore -gt 25 && $abuseipdbisPublic -eq "true" && $abuseipdbisWhitelisted -eq "false" ]]
then
printf "$remote_addr\n" >> /var/log/log.log
printf "0 AbuseIPDB detected the IP as Malicious."
else
printf "1 Unable to parse abuseIPDB output"
fi

May I ask if you would have any thoughts, regarding how to make the IP check against the abuseIPDB be done only once per session, most likely by using the SessionID set earlier?