iblocklist.com generic ipset loader for ipset v6 and v4

[redhat27]

@Twiglets Thanks for spotting the omission. You can shorten the list name to proceed for now. I'll update the script tomorrow

@Csection Since it's in github, It's easy to see the latest version. Also what changed since the last version. Also, I promise to update this thread if I update anything major that will fix something (like @Twiglets findings)

@ryzhov_al Glad to see you on this thread . I've noticed some oddities on the iblocklist data myself, some lists, like the ones maintained by TBG seems very restrictive as well (I use hotmail, and that stops working till I whitelist it)

 

[mike37]

I've uploaded a version2 and version3 of the script with a lot of extra configuration ability: Blocklists, allowlists, blacklist domains and whitelist domains, and also control inbound, outbound or both traffic. I think it may be best to keep this options separate from the original script as it may confuse some users with all those added options.

<snip great content>

THANK YOU for doing this, redhat27!!

 

[Csection]

I just meant in the code like :
#bin/sh
ver. 2 or 3.
Cause when we copy the code with a new version, we can tell if we updated it or not.
It wasn't meant to be a criticism. Just a thought.

 

[redhat27]

THANK YOU for doing this, redhat27!

@mike37 THANK YOU for your kind words and encouragement

I just meant in the code like :
#bin/sh
ver. 2 or 3.
Cause when we copy the code with a new version, we can tell if we updated it or not.

@Csection I did not take it as a criticism , and you are right, there were too many versions . I removed v3 and merged it with v2. It was getting difficult managing all of them.

Now there are just two versions. The original (and simple) one and the version2 which has a whole bunch of options to tweak. Here are the latest "v2" changes:

• The v3 of the script has now been merged to v2 and v3 will be removed shortly in the future
• Country lists from v3 are now part of v2 (you can optionally blacklist/whitelist countries) See post #32 for a (crude) example
• You can now control inbound/outbound traffic on each list independently (this was a functionality only on earlier v3)
  
• The script will now log to console if run from the command prompt (no more looking at a blank screen while the script runs)
  
• The list names of some lists have been shortened to accommodate 31 characters limit in ipset list names (Thanks @Twiglets)
  
• You can use the example blacklist-domains.txt as is to block Microsoft telemetry, Shodan and Project 25499 scanners, and then then add your own (if needed)

Please let me know if you like these changes. Feedback is always welcome and appreciated.

 

[mike37]

Greets Redhat27. If I haven't screwed this up, the whitelist and blacklist domains are not being processed under new version 2. Under old v2, a syslog entry indicated that each was processed (at the end of the ipset logs).

Also, I'm now using /mnt/sda1/ipset_lists on a flashdrive (11$ for usb3 32G sandisk at bestbuy) as recommended by Merlin. You may want to make it an alternative to /jffs/ipset_lists within the iblocklist loader script.

 

[mike37]

Also, are there some additional Ad lists somewhere? I'm blocking a lot of sites (e.g. with my selected blocks I couldn't get *this* page to properly load 'til I ran "ipset flush"), but a lot of ads are getting through......
(not a complaint about your contribution; simply fyi. If there are no additional ipset ad lists, perhaps a script that could create a .gz file from one of the many "hosts" files?)

 

[redhat27]

I'm now using /mnt/sda1/ipset_lists on a flashdrive

Just point IPSET_LISTS_DIR to /mnt/sda1/ipset_lists in your copy of the script. All the options defined up to line #350 in v2 script can be changed per users need

Also, are there some additional Ad lists somewhere?

You might want to try dns based adblocking instead of firewall based. I use that myself.

Firewall is better suited for blocking inbound traffic from hackers, bots, scanners, etc.
Ads are better blocked on outbound dns requests.

 

[p1r473]

Any way to get this to work with premium Iblock lists? (I am a subscriber)

I tried adding:
List296="Infing I-Blocklist http://list.iblocklist.com/?list=LI...rchiveformat=gz&username=MYUSERNAME&pin=MYPIN src,dst"
with the listurl, my username, and my pin, all redacted

First, had to make sure that it was less than 31 characters long. But after that, I got the error:

gunzip: invalid magic
iblocklist-loader.sh: Loaded I-BlocklistInfingSingle blocklist with 1 entries
iblocklist-loader.sh: Loaded I-BlocklistInfingCIDR blocklist with 1 entries

 

[redhat27]

The script is for free lists. The gzipped archive may be password encrypted. You can try to modify the script to gunzip with your pin, but I cannot guarantee it would work, as I've never tried it myself.

If you wget the archived file (with your username/pin url), can you gunzip without a password? If you can, the script should work.

 

[amplatfus]

Hi,

Thank you all for hard work!

Please I have to questions if you can please clarify me:
1. regarding traffic direction: src = source internet directed to my pc and dst = source my pc directed to internet (example: browsing from my pc to internet)?
2. I have confirmation messages in syslog, but how can I verify that is working? I have tried with browsing to the list page, download the archive and check browsing to random IP but I guess is not a good test.

Edit: I have found out the answers:
1.

• SRC= The source ip-address from where the packet originated
• DST= The destination ip-address where the packets was sent to

2.a) [iptables --list] And about logging activation here: https://www.snbforums.com/threads/traffic-logging-questions.31363/#post-248615
2.b) [iptables -L -v -n] source: https://www.snbforums.com/threads/feature-request-support-blocklists.11450/#post-320640

Thank you so much,
amplatfus

 

[redhat27]

Sorry I was away and couldn't respond sooner. Glad you got it sorted.

 

[amplatfus]

Hi,

I just discovered this step from Tor and Countries Block. Please, is something similar necessary in iblocklist-loader also?

Note that every time you do something on the web UI or through your android app or ios app to control your router that affects reloading the firewall rules, /jffs/scripts/firewall-start will be called, so the iptables rules that are defined outside will be wiped out. To reinstate the rules as defined by this script, you'd need to add this to your existing /jffs/scripts/firewall-start:
# Reinstate the ipset rules if they have been created already
[ "$(uname -m)" = "mips" ] && MATCH_SET='--set' || MATCH_SET='--match-set'
for ipSet in $(ipset -L | sed -n '/^Name:/s/^.* //p'); do
  case $ipSet in
   AcceptList) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j ACCEPT;;
   TorNodes|BlockedCountries|CustomBlock) iptables-save | grep -q "$ipSet" || iptables -I INPUT -m set $MATCH_SET $ipSet src -j DROP;;
   MicrosoftSpyServers) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet dst -j DROP;;
   *) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;
  esac
done

Source: https://github.com/RMerl/asuswrt-me...allation-instructions#tor-and-countries-block
Thank you!

 

[redhat27]

Yes, if your firewall is restarted, then that default case:

*) iptables-save | grep -q "$ipSet" || iptables -I FORWARD -m set $MATCH_SET $ipSet src,dst -j DROP;;

will be executed (the default being to drop both inbound/outbound traffic for the ipset). If that is not what you want, you may need to change that for your case.

 

[bayern1975]

how to run this script? i put iblocklist-loader in jffs/scripts, add sh /jffs/scripts/iblocklist-loader.sh in firewall-start and make directory in jffs/ipset_lists.......rebooted router but script didn`t run?

EDIT: i forgot something....it working now but i see just something from bluetack.....do i change in script to block all 57 lists or just 4 as set default?

admin@RT-AC3200-0000:/tmp/home/root# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  tun21  *       0.0.0.0/0            0.0.0.0/0  
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:1195
    0     0 DROP       icmp --  ppp0   *       0.0.0.0/0            0.0.0.0/0            icmptype 8
  113 12810 SECURITY_PROTECT  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22
 3545  592K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    3   151 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
  457 66250 ACCEPT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0            state NEW
 1021 94996 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 ACCEPT     2    --  *      *       0.0.0.0/0            224.0.0.0/4
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.0/4          udp dpt:!1900
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 dpt:68
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmp !type 8
   31  2629 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0  

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackProxyCIDR src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackProxySingle src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackWebexploitCIDR src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackWebexploitSingle src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackDshieldCIDR src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackDshieldSingle src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackSpiderCIDR src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set BluetackSpiderSingle src
    4   192 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set privacy-filter_ipv4 src,dst reject-with icmp-port-unreachable

 

[redhat27]

Your posted output seems normal. Give it some time to accumulate blocked stats. My experience was that BluetackSpider list gets quite a few hits over time

 

[bayern1975]

Your posted output seems normal. Give it some time to accumulate blocked stats. My experience was that BluetackSpider list gets quite a few hits over time

do i change in script to block all 57 lists or just 4 as set default? what is best setup for best security?

 

[amplatfus]

Hi again. I do not understand why I have 2 entries for BlacklistDomains. Please, is it normal? This output is after clean restart. Thank you!

rooter@root:/tmp/home/root# iptables -S
-A FORWARD -m set --match-set WhitelistDomains src,dst -j ACCEPT
-A FORWARD -m set --match-set BlacklistDomains src,dst -j DROP
-A FORWARD -m set --match-set BlacklistDomains src,dst -j DROP

 

[bayern1975]

i have lot`s of duoble writing too?

-A FORWARD -m set --match-set BluetackMicrosoftCIDR src -j DROP
-A FORWARD -m set --match-set BluetackMicrosoftSingle src -j DROP
-A FORWARD -m set --match-set BluetackMicrosoftCIDR src -j DROP
-A FORWARD -m set --match-set BluetackMicrosoftSingle src -j DROP
-A FORWARD -m set --match-set BluetackProxyCIDR src -j DROP
-A FORWARD -m set --match-set BluetackProxySingle src -j DROP
-A FORWARD -m set --match-set BluetackProxyCIDR src -j DROP
-A FORWARD -m set --match-set BluetackProxySingle src -j DROP
-A FORWARD -m set --match-set BluetackWebexploitCIDR src -j DROP
-A FORWARD -m set --match-set BluetackWebexploitSingle src -j DROP
-A FORWARD -m set --match-set BluetackWebexploitCIDR src -j DROP
-A FORWARD -m set --match-set BluetackWebexploitSingle src -j DROP
-A FORWARD -m set --match-set BluetackDshieldCIDR src -j DROP
-A FORWARD -m set --match-set BluetackDshieldCIDR src -j DROP
-A FORWARD -m set --match-set BluetackDshieldSingle src -j DROP
-A FORWARD -m set --match-set BluetackSpiderCIDR src -j DROP
-A FORWARD -m set --match-set BluetackSpiderCIDR src -j DROP
-A FORWARD -m set --match-set BluetackSpiderSingle src -j DROP
-A FORWARD -m set --match-set BluetackSpiderSingle src -j DROP
-A FORWARD ! -i br0 -o eth0 -j Blacklist
-A FORWARD ! -i br0 -o eth0 -j Blacklist
-A FORWARD -i eth0 -m state --state INVALID -j Blacklist
-A FORWARD -i eth0 -m state --state INVALID -j Blacklist
-A Blacklist -m state --state NEW -j SET --add-set Blacklist src
-A Blacklist -m state --state NEW -j SET --add-set Blacklist src
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j RETURN
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec -j RETURN
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -j DROP

 

[redhat27]

do i change in script to block all 57 lists or just 4 as set default? what is best setup for best security?

You can choose any number of list. The original iblocklist-loader has 37 common lists, the extended version has 295 lists that include countries and more options. In both cases, just choose the lists you are interested in and modify the BLOCKLIST_INDEXES= line to include your chosen lists. Also see the first post in this thread I attempted to explain this a bit there.

What is best for your security is up to you to decide

 

[redhat27]

Hi again. I do not understand why I have 2 entries for BlacklistDomains. Please, is it normal? This output is after clean restart. Thank you!

rooter@root:/tmp/home/root# iptables -S
-A FORWARD -m set --match-set WhitelistDomains src,dst -j ACCEPT
-A FORWARD -m set --match-set BlacklistDomains src,dst -j DROP
-A FORWARD -m set --match-set BlacklistDomains src,dst -j DROP

No, it is not normal to have BlacklistDomains listed twice. I suspect the script may be called twice or another script is using the same name. Can you verify what scripts are called on startup? The syslog may give some indication when and what scripts are executing when.