How to Dynamically Ban Malicious IP's using IPSet (Martineau version)

[Martineau]

So I trying to understand the purpose of the script better. Is it that the script looks at the syslog for packets already being dropped by the firewall and puts them in a blacklist ipset

So what is the benefit gained?

Really?

I could ask the same question of you...if the firewall by default DROPs everything inbound why do we need your TOR / Country blocking script?

You can stop reading here if you suspect TL;DR may apply!

So, being paranoid about basic vendor provided firewall security, I firmly believe it is never a waste of time to be proactive and vigilant to utilise efficient tools/techniques to ensure nothing is left to 'chance' when there are daily threats from the internet.

However, as an explanation, here is one scenario:

Suppose I am hosting a service say a website on the standard port 80 or say port 54321 that requires a logon ID and password combo for UK based family and friends.

Where in the above scenario does the default DROP rule get triggered?

Now I can't use your (UK) country blocking script, that would be silly wouldn't it? so how can I differentiate between family members who legitimately should be allowed to attempt access (even after a frustrating failed login due to finger trouble with their password etc.) and some little thievin' scroat or bot?

Q. Could the 'static' lists from say ipdeny help me in anyway? - possibly, it depends.

So I'm not saying that the dynamic IPSET blocking script is foolproof, but at least it gives me the opportunity to be proactive, and possibly take further action i.e. fine tune the criteria to decide if I should add/retain an IP in the dynamic Blacklist.

i.e. why would any of my family/friends want to try and access say all three ports 22,23,25 etc.?... Sod's Law says eventually one of them may be curious but they are all made aware of the consequences and accept that they could embarrassingly find themselves on the naughty step!

 

[redhat27]

I'm so very sorry if I sounded brash. I just wanted to understand better.

if the firewall by default DROPs everything inbound why do we need your TOR / Country blocking script?

Yes, but for ports that you open specifically to let in outside traffic, you'd still need to filter from sources you do not want, right?

Where in the above scenario does the default DROP rule get triggered?

If you have opened ports to let in traffic, the DROP would not get triggered... If country block was used in conjunction, it would filter on IPs that are assigned for that country (not going into spoofed IP discussion)

Now I can't use your (UK) country blocking script, that would be silly wouldn't it?

I did not get this part at all. Maybe I am a bit thick headed. If you want UK based members to login, why would you ever block UK in country block?

it gives me the opportunity to be proactive, and possibly take further action i.e. fine tune the criteria to decide if I should add/retain an IP in the dynamic Blacklist

I agree on this. Also you have nice archive of IPs that actually tried to access

If you are concerned on access on specific ports that are NOT open, the built in FW will already block that. So I am still not very clear as to the benefit as the script is harvesting IPs that are already being dropped.

In any case, I wanted to apologize if I appeared to belittle this script. I want to stress that was not my intent.

 

[Martineau]

@Adamm
It would need some other ways to scan the syslog

Neither @Adamm's nor my script physically needs to scan Syslog in order to actually populate the IPSET - the DROP messages are simply a tedious 'noise' side-effect and @Adamm's script removes the DROP messages from Syslog in real-time leaving no evidence.

However as posted in #144, my version currently retains the DROP messages as I run a daily Report against the DROP messages at which time they are then physically removed.

 

[Martineau]

IPSET_Block.sh updated to v3.04

https://www.snbforums.com/threads/h...ious-ips-using-ipset-martineau-version.38748/

Changes:

1. The rewritten v3 method no longer uses the 'logdrop' chain, and allows the script to dynamically ban IPs without forcing the modification of the GUI Firewall-> 'Logged packets type' setting.

i.e. the user now retains complete control so 'Logged packets type' can be either 'None' or 'Accepted' etc.

2. The new v3 script can still write custom 'Block IN=' messages to syslog for Hacker reporting - use 'nolog' arg to suppress the creation of the messages. (see help)

RT-AC68U kern.warn kernel: Block IN=eth0 OUT= MAC=xx:xx:xx........ SRC=xxx.xxx.xxx.xxx

3. The v3 script allows the user to specify an auto-expiry time for banned entries. (see help)


e.g. banned IPs will be removed from the Blacklist IPSET after 12 hrs 34 mins 56 secs

./IPSET_Block.sh

(IPSET_Block.sh): 17938 v3.03 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....

 Summary Blacklist: 1144 Successful blocks! (13443 IPs currently banned - 11 added since: Apr 21 16:41 , Entries auto-expire after 12:34:56)

NOTE: There is no way for the script to know the true number of reported 'added' entries when using the auto-expire feature

i.e. if the script status report says '1 added', it could be that since the last status request, 100 new banned IPs were physically added, but also 99 expired entries were physically deleted.

(The timeout feature is only available on routers with IPSET v6.)

Spoiler: IPSET_Block Help/ Example Usage:

./IPSET_Block.sh help

#======================================================================================================= © 2016-2017 Martineau, v3.01
#
# Dynamically block unsolicited access attempts using IPSETs. Useful if you have opened ports >1024 as hopefully hackers will
#             start their attempts at the more common ports e.g. 22,23 etc. so will be banned BEFORE they reach your port!
#             NOTE: For ARM routers (IPSET v6.3) Blacklist entries are retained for 7 days unless arg HH:MM:SS is specified/hard-coded)
#
#     IPSET_Block   [help | -h] | [status [list]] [reset] [delete] [ban {'ip_addr'}] [unban {'ip_addr'}] [restore] [nolog]
#                               { init [reset] ['hh:mm:ss'] [method1] }
#
#     IPSET_Block
#                   Displays the number of currently banned I/Ps and the number of banned IPs added since the last status request:
#                            e.g. 'Summary Blacklist: 12918 IPs currently banned. 40 New IP's banned since: Apr 11 12:34 Apr 16 15:27 (Entries auto-expire after 12:34:56)'
#     IPSET_Block   status list
#                   Display the contents of IPSETs Whitelist & Blacklist - beware there could be a lot!!!
#     IPSET_Block   reset
#                   Temporarily flush the IPSET Blacklist (It will be restored @BOOT or manually using the restore cmd)
#     IPSET_Block   restore
#                   Restore the IPSETs Whitelist & Blacklist from the current saved IPSETs.
#                   (If 'delete' was used then you need to clone the 'backup' file before attempting the restore!)
#     IPSET_Block   ban 12.34.56.7
#                   Adds 12.34.56.7 to IPSET Blacklist
#     IPSET_Block   unban 12.34.56.7
#                   Removes 12.34.56.7 from IPSET Blacklist
#     IPSET_Block   delete
#                   Permanently flush the IPSET Blacklist (It cannot be restored @BOOT or using the restore cmd)
#     IPSET_Block   init
#                   If 'IPSET_Block.config' exists it will be used to restore IPSETs Blacklist and Whitelist,
#                      otherwise the IPSETs are created empty - same as if 'init reset' was specified to override the auto-restore
#     IPSET_Block   init reset 12:34:56 nolog
#                   Empty IPSETs will be created with any added Blacklist entries auto-expiring after 12 hrs 34 mins and 56 secs!
#                         (default expiry time is 168:00:00 = 7 Days)
#                          NOTE: No Syslog 'Block =' messages will be generated.
#
# /jffs/scripts/init-start
#      /usr/sbin/cru a IPSET_SAVE   "0 * * * * /jffs/scripts/IPSET_Block.sh save"    #Every hour
#      /usr/sbin/cru a IPSET_BACKUP "0 5 * * * /jffs/scripts/IPSET_Block.sh backup"  #05:00 every day
#
# /jffs/scripts/firewall-start
#      /jffs/scripts/IPSET_Block.sh init nolog
#
# NOTE: Whitelist will be automatically populated with local LAN subnet, but VLANs will need to be added manually e.g. 10.0.0.0/8 etc.
#
# Credit @adamm https://www.snbforums.com/threads/how-to-dynamically-ban-malicious-ips-using-ipset-firewall-addition.16798/#post-115872
# Also Very good examples of using IPSETs for blocking dynamically! https://forums.gentoo.org/viewtopic-t-863121.htm

 

[HardCat]

@Martineau - I have installed v3.01 of your script and am happy to report it is working well. However I did need to make a small change due to the fact that the label on my usb stick was different from yours. When I made the change I did notice that in 2 places in the script your label "AC68U" was hard coded and would like to suggest where the following is in the script: (2 instances)

XRETAIN_SECS=$(grep "^create Blacklist" /mnt/RT-AC68U/IPSET_Block.config | grep -oE "timeout.*" | cut -d" " -f2)

Perhaps change to:

XRETAIN_SECS=$(grep "^create Blacklist" $DIR"/IPSET_Block.config" | grep -oE "timeout.*" | cut -d" " -f2)

if that would be the correct syntax.

Thanks...

 

[Martineau]

@Martineau I did notice that in 2 places in the script your label "AC68U" was hard coded and would like to suggest where the following is in the script: (2 instances)

XRETAIN_SECS=$(grep "^create Blacklist" /mnt/RT-AC68U/IPSET_Block.config | grep -oE "timeout.*" | cut -d" " -f2)

Perhaps change to:

XRETAIN_SECS=$(grep "^create Blacklist" $DIR"/IPSET_Block.config" | grep -oE "timeout.*" | cut -d" " -f2)

@HardCat Abject apologies..

In my defense I was taught

"ALL bug-free code is trivial"

therefore by definition, any of my code that has bugs must be exceedingly non-trivial...well that's my excuse!

I normally do label my USB sticks with the model name of the router, in the hope this stops me sticking a USB stick into the wrong router.

Indeed your suggestion to replace the rogue hard-coded reference (which actually was originally '/mnt/$MYROUTER' until I obviously did a dumb cut'n'paste from the command prompt during testing) with '$DIR' is correct, but actually the line doesn't report on the intended metric.

So to correct the intended reporting, here is what the re-written code-line should actually be - hopefully you would agree?

XRETAIN_SECS=$(ipset list Blacklist | head -n 4 | grep -E "^Header" | grep -oE "timeout.*" | cut -d" " -f2)

Anyway many thanks for 'beta' testing my shonky code...I've changed the line as indicated above and published v3.02 if anyone is brave enough to test.

 

[HardCat]

@HardCat Abject apologies..

In my defense I was taught

"ALL bug-free code is trivial"

therefore by definition, any of my code that has bugs must be exceedingly non-trivial...well that's my excuse!

I normally do label my USB sticks with the model name of the router, in the hope this stops me sticking a USB stick into the wrong router.

Indeed your suggestion to replace the rogue hard-coded reference (which actually was originally '/mnt/$MYROUTER' until I obviously did a dumb cut'n'paste from the command prompt during testing) with '$DIR' is correct, but actually the line doesn't report on the intended metric.

So to correct the intended reporting, here is what the re-written code-line should actually be - hopefully you would agree?

XRETAIN_SECS=$(ipset list Blacklist | head -n 4 | grep -E "^Header" | grep -oE "timeout.*" | cut -d" " -f2)

Anyway many thanks for 'beta' testing my shonky code...I've changed the line as indicated above and published v3.02 if anyone is brave enough to test.

There is one more instance at line 311 of v3.02 should that also be changed?

 

[Martineau]

There is one more instance at line 311 of v3.02 should that also be changed?

There are no instances of 'RT-AC68U' in v3.02???

 

[HardCat]

There are no instances of 'RT-AC68U' in v3.02???

My apologies for not explaining further. I was referring to how XRETAIN_SECS gets populated at line 311 versus line 449?

 

[Martineau]

My apologies for not explaining further. I was referring to how XRETAIN_SECS gets populated at line 311 versus line 449?

Ahh I see.

Well like all lazy programmers I naively assume that the ipset restore command on Line 310 always physically works!

Line:310  ipset $RESTORE  < $DIR/IPSET_Block.config
Line:311  XRETAIN_SECS=$(grep "^create Blacklist" $DIR/IPSET_Block.config | grep -oE "timeout.*" | cut -d" " -f2){/CODE]

So in 99.99999% of cases there is no need to change Line 311 as the restored IPSET should be configured from the $DIR file, but you are again correct - one day Line 311 will report incorrect data

Line 449 (originally cloned from Line 311) fixes the scenario where you may have changed your mind about the IPSET member expiry time

i.e. suppose you decide to manually issue the following at the command prompt

./IPSET_Block.sh init reset 36:00:00
./IPSET_Block.sh init reset 48:00:00
./IPSET_Block.sh init reset 120:00:00

./IPSET_Block.sh

(IPSET_Block.sh): 17938 v3.03 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....

 Summary Blacklist: 1144 Successful blocks! (13443 IPs currently banned - 11 added since: Apr 21 16:41 , Entries auto-expire after 120:00:00)

..surely you want the status report to reflect what 'timeout' value is currently in the created IPSET (5 days) rather than from a config file that is no longer in use!

 

[Martineau]

@Martineau mind helping this user ?

I have dumped a copy of my script

IPSET_Block.sh v3.04

to

https://github.com/MartineauUK/IPSET_Block/blob/master/IPSET_Block.sh

Feel free to make a start to include it in the IPSET Wiki page, and hopefully I'll have time to add additional Wiki details when time permits.

NOTE: v3.04 should now allow users of MIPS/ARM routers that are still running IPSET v4 to successfully use my script.

 

[swetoast]

kewl ill add it tnx for the git

 

[swetoast]

https://github.com/RMerl/asuswrt-merlin/wiki/Using-ipset

https://github.com/RMerl/asuswrt-me...on-instructions#dynamically-ban-malicious-ips

feel free to update the missing info.

 

[Csection]

I loaded Version 3.04 and I am getting some errors:
1) Iptables: No chain/target/match by that name
2) Ipset V6.29 The set with the given name does not exist.

 

[Martineau]

I loaded Version 3.04 and I am getting some errors:
1) Iptables: No chain/target/match by that name
2) Ipset V6.29 The set with the given name does not exist.

Ahh, I have confused you with another user that was having problems where they have IPSET v4.5 such as MIPS routers or say an ARM router such as RT-AC56U running an old version of firmware such as 380.59.

So, whilst I suggest you try v3.04, this version only contains changes to try and retain backward compatibility with older firmware.

Did you issue the following:

cd /jffs/scripts

./IPSET_Block.sh   init   nolog

./IPSET_Block.sh

 

[Csection]

Ahh, I have confused you with another user that was having problems where they have IPSET v4.5 such as MIPS routers or say an ARM router such as RT-AC56U running an old version of firmware such as 380.59.

So, whilst I suggest you try v3.04, this version only contains changes to try and retain backward compatibility with older firmware.

Did you issue the following:

cd /jffs/scripts

./IPSET_Block.sh   init   nolog

./IPSET_Block.sh

No! I didn't do that.
I just ran it from the command line.
I have not yet set up init-start or firewall-start. I wanted to make sure it would work first.
I do have a problem that when I copy the script from git, it does not copy correctly(Character overrun, lines too long.). I have to go through and try and fix the code.
I'll try the code you suggest and see if it makes a difference.

 

[Csection]

Ahh, I have confused you with another user that was having problems where they have IPSET v4.5 such as MIPS routers or say an ARM router such as RT-AC56U running an old version of firmware such as 380.59.

So, whilst I suggest you try v3.04, this version only contains changes to try and retain backward compatibility with older firmware.

Did you issue the following:

cd /jffs/scripts

./IPSET_Block.sh   init   nolog

./IPSET_Block.sh

Ok! I tried that and it did list ips in my syslog, but I am getting an error on the command line saying "Line 134 not found".
I don't know what file it is referring to.

 

[Csection]

The code right before this line is :
if [ "$(echo $@ | grep -c 'nolog')" -qt 0 ]; then
NOLOG=1 #Suppress "block=" from syslog
fi
fi"
The init nolog seemed to run without error, but just running ./ipset_block.sh gives the line 134 not found msg.

 

[Martineau]

I do have a problem that when I copy the script from git, it does not copy correctly(Character overrun, lines too long.). I have to go through and try and fix the code.

I suspect the fact that you have to 'try and fix the code' due to the download being 'corrupt' is the root cause of your current issues.

I have just created a test6.sh from the script hosted here

https://pastebin.com/zQ3KEe8P

and it runs fine

./test6.sh init nolog

(test6.sh): 21788 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....
(test6.sh): 21788 IPSET restore from '/tmp/mnt/RT-AC68U/IPSET_Block.config' starting.....
(test6.sh): 21788 Dynamic IPSET Blacklist banning enabled.
(test6.sh): 21788 Hacker Port Activity report scheduled every 06:05 daily

 Summary Blacklist: 0 Successful blocks! ( 3427 IPs currently banned - 0 added since: Apr 25 19:19 ), Entries auto-expire after 168:00:00 hrs

 ./test6.sh

(test6.sh): 21957 v3.04 © 2016-2017 Martineau, Dynamic IPSET Blacklist banning request.....

 Summary Blacklist: 0 Successful blocks! ( 3427 IPs currently banned - 0 added since: Apr 25 19:22 ), Entries auto-expire after 168:00:00 hrs

Please try creating the script from pastebin rather than the GitHub version.

EDIT: Just created a test0.sh script from the GitHub version and that too runs correctly.

 

[HardCat]

I pulled v3.04 from GitHub this morning and it runs fine here.