iblocklist.com generic ipset loader for ipset v6 and v4

[redhat27]

Since I had initially started to port the peer guardian implementation in the asus wiki page to ipset v6, I noticed that simply changing the v4 commands for v6 does not work, as found by @amigohd

This led me to use awk to separate out IP ranges to CIDR notation and have two sets created for ipset v6 implementation: One for single IPs and one for CIDR.
Ipset v4, still uses iptreemap hashtype, so there is no change in that area.

This, and the fact that all free downloads from the iblocklist site appear to obey the same format, let me to create a generic ipset loader for data provided in zipped IP range format.

So the script has a one line control of all the target lists from iBlocklist you'd like to use:

For example, if you were to implement the current peerguardian, for example, you'd use

BLOCKLIST_INDEXES="2"

or

BLOCKLIST_INDEXES="2 11"

You can block traffic from Tor nodes and other anonymizers by including List10:

BLOCKLIST_INDEXES="10"

Want to block anyone from playing Second Life? Block Linden labs using (with v2 or v3 script)

BLOCKLIST_INDEXES="49"

Known hackers with dshield using List15

BLOCKLIST_INDEXES="15"

Webmasters can block hostile spiders using List13

BLOCKLIST_INDEXES="13"

Hijacked and spam can be blocked using List14

BLOCKLIST_INDEXES="14"

You can combine as many lists as you'd like on the BLOCKLIST_INDEXES
The possibilities are quite large as what you can block

Here is the version2 script in my git repo. It offers a more lists and options over version 1. Feel free to use whichever one you like.

The latest v2 script has undergone several changes. Please see post #44 for an update summary

Please feel free to test it and provide feedback. If the script is useful enough, I'll include it in the wiki Its now in the wiki

 

[redhat27]

Anyone tested this yet? I was not able to post the script on the forum due to size limits.

The script is here. I would appreciate it very much if someone tests the script either as is or changing the BLOCKLIST_INDEXES to suit your selection. If you search "block list" on google, the iblocklist site is the top result, so I really think this script could be useful, especially to users of ipset v6.x

Any suggestions for improvements are welcome

 

[Simmz]

So I know I'm late to the party but I just tried this and it failed to load under ASUSWRT-Merlin/Xwrt-Vortex 380.65.

I read that the kernel module names changed in ipset v6 so I checked what the script called VS what the kernel has.

I changed "ip_set_nethash" to "ip_set_hash_net" & "ip_set_iphash" to "ip_set_hash_ip" in the v6 section and all is well.

Works great so far, maybe a little to go as Pandora stopped streaming once the script loaded but I'm looking into that

Just thought I would create an account so at least you got SOME feedback.

Nice work and thanks!

Edit: Not sure if it matters to you but I'm running it on a r7000

 

[redhat27]

Just thought I would create an account so at least you got SOME feedback.

Thank you! Lack of any response had me thinking this wasn't so useful after all. And welcome to the forum!

Yes, I do not know the kmods of Xwrt-Vortex so I'm glad it worked for you after you switched the ones in the script. Can you post your output for uname -m for your router?

Regarding Pandora stopping, its easy to test if an IP is in an ipset list. Which blocklist(s) are you using?

Thanks again for testing the script

 

[amigohd]

With either
Primary-Threats TBG or
level1 Bluetack
Apple Services and iCloud Services are blocked. How to unblock/remove Apple entries in the created .txt File? Or more global: How to determine IP adresses/ranges of a specific domain and delete them from the .txt before loading them with ipset?

 

[redhat27]

How to determine IP adresses/ranges of a specific domain and delete them from the .txt before loading them with ipset?

That part is easy: You can use the entware hostip utility (opkg install or part of DNSCrypt proxy install)
So for example, if you wanted to get the list of IPs for say login.live.com, you'd run hostip login.live.com you can even get AAAA records with a -6 option for that host/domain

That script should allow for some special handling (and thank you for voicing that) How would you want it? I can have it to process an allowed domain list, for example

 

[amigohd]

An allowed domain list would be the simplest way I guess. i.e you place a file called "whitelist" to /jffs with content:

apple.com
icloud.com
whatever.com

and the scripts does the rest. I do know hostip and I have installed DNSCrypt as well but I am no linux/shell guru, so don't know how to automate it (greping lines, ips etc. and delete them from the *list*.txt before ipset). It would be great to have that option of a whitelist within your script, if it's simple enough for people, like me

 

[redhat27]

It is difficult to remove single IP(s) in a ipset CIDR list. Does anyone have any ideas on how to accomplish this?

One way I think could be to have a separate iptables rule of whitelist that would be checked before the rule that checks for blocklist

 

[Simmz]

Thank you! Lack of any response had me thinking this wasn't so useful after all. And welcome to the forum!

Yes, I do not know the kmods of Xwrt-Vortex so I'm glad it worked for you after you switched the ones in the script. Can you post your output for uname -m for your router?

Regarding Pandora stopping, its easy to test if an IP is in an ipset list. Which blocklist(s) are you using?

Thanks again for testing the script

As I usually tend to do things I went a little over board with the blocklists at first. I'm currently blocking with list (2 8 9 11 14 29)

uname -m returns "arm71"

 

[Simmz]

It is difficult to remove single IP(s) in a ipset CIDR list. Does anyone have any ideas on how to accomplish this?

One way I think could be to have a separate iptables rule of whitelist that would be checked before the rule that checks for blocklist

I'm sure there is a WAY cleaner way of doing this but for the time being I just added a iptables rules to the bottom of the script. As an example..

iptables -I FORWARD 1 -s 208.85.40.0/21 -j ACCEPT

That is the address range of PANDORA which fixed my streaming issue.

Again, I'm sure there is a much more appropriate was to accomplish this but my brain shuts down without music so that was priority one!

 

[redhat27]

@Simmz @amigohd Can you check out the updated script in my repo? It now handles whitelist domains if the file exists. I've uploaded a sample file as well.

I changed "ip_set_nethash" to "ip_set_hash_net" & "ip_set_iphash" to "ip_set_hash_ip" in the v6 section and all is well.

I would hate to have you make this change over and over if you use the script in github, can you find a way where I can identify which router has which kmods? I myself have a mips router with ipset 4.5

 

[redhat27]

@Simmz I think I had my kernel modules bad from the start in the v6 section. I've updated the script. You should now be able to use the script as is

@amigohd can you verify that the updated kernel modules works for you as well?

 

[Simmz]

@Simmz I think I had my kernel modules bad from the start in the v6 section. I've updated the script. You should now be able to use the script as is

The updated script and whitelist functionality work like a charm.

I do have one questions though. The script re-downloads the block-lists every time it's ran regardless of age. This causes a 4 min delay which I'm fine with, but isn't it supposed to use the cache unless it's expired? It's probably something on my end since I posses just enough skill to REALLY damage things.

 

[swetoast]

reviewed your code a bit and i saw that hostip is needed for the whitelist, you can do like i did on privacy-filter and use traceroute also less dependencies and it works out of the box for those that dont have entware

cat $WHITELIST_DOMAINS_FILE | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> $WHITELIST_FILE

in anycase its an suggestion

 

[redhat27]

reviewed your code a bit and i saw that hostip is needed for the whitelist, you can do like i did on privacy-filter and use traceroute also less dependencies and it works out of the box for those that dont have entware

cat $WHITELIST_DOMAINS_FILE | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> $WHITELIST_FILE

in anycase its an suggestion

Thanks for the suggestion. I removed hostip (entware) dependency in the script.

I did not understand how traceroute can be used to get all the IPs that a host might resolve to. For example if you do hostip login.live.com you get several IPs:

Spoiler: hostip login.live.com

admin@RT-AC66R-D700:/tmp/home/root# hostip login.live.com
131.253.61.82
131.253.61.96
131.253.61.68
131.253.61.100
131.253.61.98
131.253.61.80
131.253.61.66
131.253.61.64

Now sure how to achieve that with traceroute.

I opted to use nslookup, that is bundled with busybox (usually available in /usr/bin/nslookup)

 

[redhat27]

I do have one questions though. The script re-downloads the block-lists every time it's ran regardless of age. This causes a 4 min delay which I'm fine with, but isn't it supposed to use the cache unless it's expired?

The purpose of the USE_LOCAL_CACHE=Y and LISTS_SAVE_DAYS=n is that once the blocklist gzip is downloaded, it will not download it again for number of days specified in LISTS_SAVE_DAYS
I believe if you check the timestamp of the .gzip files, you'll see that there are not being re-downloaded even after your router is rebooted (if you used USE_LOCAL_CACHE=Y)

The 4 minute delay you experienced is due to the ipset being processed and loaded in memory again, not due to he download.

If the plan is to run the script in cron at a higher frequency than the number of days specified in LISTS_SAVE_DAYS, I would suggest you set USE_LOCAL_CACHE=N

HOWEVER: If the intent is to not reload the blocklist at all, if the blocklist has already been loaded, I can rework the script to make that happen (if USE_LOCAL_CACHE=Y)
I think that is a good idea.

 

[swetoast]

Thanks for the suggestion. I removed hostip (entware) dependency in the script.

I did not understand how traceroute can be used to get all the IPs that a host might resolve to. For example if you do hostip login.live.com you get several IPs:

Spoiler: hostip login.live.com

admin@RT-AC66R-D700:/tmp/home/root# hostip login.live.com
131.253.61.82
131.253.61.96
131.253.61.68
131.253.61.100
131.253.61.98
131.253.61.80
131.253.61.66
131.253.61.64

Now sure how to achieve that with traceroute.

I opted to use nslookup, that is bundled with busybox (usually available in /usr/bin/nslookup)

Kewl, ill check it out one your done tinkering with it

 

[redhat27]

If the intent is to not reload the blocklist at all, if the blocklist has already been loaded, I can rework the script to make that happen

Made it happen

Kewl, ill check it out one your done tinkering with it

Done tinkering before I posted #15

 

[Simmz]

HOWEVER: If the intent is to not reload the blocklist at all, if the blocklist has already been loaded, I can rework the script to make that happen (if USE_LOCAL_CACHE=Y)
I think that is a good idea.

Correct me if I'm wrong (which I probably am) but the WHITELIST is read, then loaded into iptables only when the full script runs correct? My issue currently is that I continue to discover domains that needed to be added to the whitelist. Which means in order for that change to take effect I have to re-run the full script... 4 mins later I can see if it worked.

I'm not complaining as I DO appreciate what you've done so far. I'm only asking if the blocklists are already in memory do we "really" need to reload them? Since the whitelist is first in the chain modifications could be made without affecting the rest of the process.

Not sure if I explained that correctly. Let me know if I need to re-think how to explain it.

On a side note I've continued testing the script as you update it. In doing the following popped up with the latest version (using nslookup in place of hostip).

ipset v6.29: Syntax error: cannot parse 2620:106:e003:f00e::63: resolving to IPv4 address failed
ipset v6.29: Syntax error: cannot parse www.v6.pandora.com: resolving to IPv4 address failed
Warning: www.pandora.com resolves to multiple addresses: using only the first one returned by the resolver.
ipset v6.29: Element cannot be added to the set: it's already added
Warning: www.pandora.com resolves to multiple addresses: using only the first one returned by the resolver.
ipset v6.29: Element cannot be added to the set: it's already added

Due to the above my music has stopped....again. :/ At least I have enough skill to fix that one with a quick iptables rule.

 

[Simmz]

So.... the above post is now useless. Yeah I would say you reworked it. Total run time was 1.28 seconds NICE !!!!