iblocklist.com generic ipset loader for ipset v6 and v4

[p1r473]

I agree, that that is not a foolproof way to determine if a lot of sites are blocked, but you only need to unblock your favourite sites. Also testing this addition to whitelistdomains should not involve rebooting your router or anything that takes too long:

After adding the domain to the whitelist domains file, you just manually run the script. It will NOT process the already loaded lists if USE_LOCAL_CACHE=Y, it will just re-process the whitelistdomains (and blacklistdomains if you are using v2). The turnaround time should be fairly quick.

@redhat27 so I am trying to figure out the whitelist, maybe you can give some insight please.
In my ssh console:

iblocklist-loader.sh: Added WhitelistDomains (9 entries)

So a few questions
1. Why did it add 9 entries when I only added 4? The code comments state I can do inline comments, is it trying to do inline comments as domains?
2. I did manage to get this error yesterday:
./iblocklist-loader.sh: line 515: syntax error: unexpected "(" (expecting "fi")
3. Do you have any idea why the script is making 2 septerate /ipset_lists/ folders?

Here is the contents of /jffs/ipset_lists/whitelist-domains.txt. Is there a way to get inline comments not counted as domains, so it shows up as only 4 domains whitelist, and not 9??

speedtest.net #29
bbc.co.uk #33
teamviewer.com #2
eotugame.com #premium malicious by squidblacklist

Also thanks for letting me know I can just rerun the script, I kept turning the firewall off then on to test this

 

[redhat27]

1. Why did it add 8 entries when I only added 4? The code comments state I can do inline comments, is it trying to do inline comments as domains?

No, some of the domains resolve to multiple IP addresses. Try this on your router:
nslookup outlook.com
nslookup yahoo.com
you should see multiple addresses. It does not process the comments.

2. I did manage to get this error yesterday:
./iblocklist-loader.sh: line 515: syntax error: unexpected "(" (expecting "fi")

I do not know what to say to that... The script does not have 515 lines maybe you have a bad copy? Can you get the script again?

 

[p1r473]

I added a few comments on new lines so the line count will be off. Ill try to keep an eye out for the error again to get the correct line number next time it fires.

Sorry I added a third question above:
Do you have any idea why the script is making 2 separate /ipset_lists/ folders? Both folders have same contents but listed twice in both putty and winscp

 

[redhat27]

There should not be 2 folders. Please go to the router /jffs directory and see if that is indeed the case. Maybe there is a space character? router OS would prevent 2 identical directories.
do a add a -al parameter to ls on router
I would advise move the contents to the correct directory and delete the other. Check where are files are present. That directory is the one that should be kept

Edit: type cd /jffs/ipset_li and then hit tab a few times... The bash autocomplete should show the 2 directories, and if there is a space in one of them

I also added your whitelistdomain entries to github

 

[p1r473]

2 folders, with different modification dates. but, I only added the whitelist to one folder but it shows up in both. So they are somehow duplicating.

Also I fixed up the whitelist, please check for my pull request on Git

 

[p1r473]

I dont see a space... could it be a \n? Somehow the script made me both folders and the contents are the same (whitelist is in both)

 

[redhat27]

I cannot think of anything other than a corrupted filesystem if there are no differences in the directory names in anyway... . The OS should not allow identical names

Can you try to delete the one dated 4/26 from the GUI application that you are using?

EDIT: merged... Thanks for your contribution

 

[p1r473]

I manually deleted both folders in putty using rmdir, and reran the script, and the duplicates don't seem to have return. Yay!

 

[swetoast]

since you added telemetry blocking you could atleast have the decency to tell your users that it renders privacy-filter useless but i guess that too much to ask.

in anycase i added it on the privacy-filter thread so thx for the heads up as usual really respectful

 

[redhat27]

I have not used your privacy-filter script myself. This iblocklist-loader script has had a Blacklist-domains file since March (that is given as an example) that blocks Microsoft telemetry, Shodan and Project 25499 scanners. I've included the sources where I got the list from and I have copy pasted them verbatim.

There are many instances of scripts that achieve similar functionality in the forums, and even more on the internet.

The country block has tor/telemetry block, and so does iblocklist-loader
The forum has more than one ways to block ads, and so is the case for loading malware lists

If you plan to get upset or pick a fight with every script/solution that remotely resembles your work, then that is your issue, not mine.

 

[swetoast]

not picking a fight just informing your users since you generally tend to not give not care about that. so now its done, ive added that your script and mine are incompatible at the wiki so there is nothing left to discuss

 

[redhat27]

not picking a fight

really?

so there is nothing left to discuss

amen to that!

 

[amplatfus]

Please, this method blocks access to open ports too? For example FTP port 21. Thanks!

Sent from my ONE A2003 using Tapatalk

 

[redhat27]

Source blocking with ipset match has nothing to do with specific ports. It blocks all port on a match. If you want to block port 21 for all cases, you should not have it open.

 

[Jack Yaz]

Can this be run on cron, or is that inadvisable?

 

[redhat27]

Can this be run on cron, or is that inadvisable?

Sure, it can be run from cron. If you select a list that is dynamic in nature, you would want to run it in cron. In that case just make USE_LOCAL_CACHE=N, and you should be all set to create a cru entry to run at a frequency you desire

 

[Jack Yaz]

I wasn't sure if after the cache expired if the script re ran at all.so if I set cache to 7 days, I'd have the script run every 8, for example. I don't intend on restarting router unless i have to!

 

[redhat27]

LISTS_SAVE_DAYS value has no effect when USE_LOCAL_CACHE=N. Only when USE_LOCAL_CACHE=Y, then that will come into play.

If you have USE_LOCAL_CACHE=N, the data will be downloaded from iblocklist.com site on every run. This is ideal when scheduled in cru

If you have USE_LOCAL_CACHE=Y, then the data will normally not be downloaded if the data file exists locally. It will only download the site data again if the file date-time stamp is more than the days defined in LISTS_SAVE_DAYS. So if LISTS_SAVE_DAYS=10, and you reboot your router many times within that time it will not re-download the files. It will only download once the date on files is older than 10 days. So if your router is rebooted say 15 days after the date the files were downloaded, it will still re-download.

 

[shooter40sw]

Hi @redhat27 Im testing this script now, and when I use this one, for example blocking proxy, list 10, the onionrotuer list 39 and china country list 110, they all got to the forward chain of iptables, but in your other script blocking telemetry , tor nodes and countries they go to the input chain, so if an IP from china is scanning IPs and there is a service open like ssh or whatever, will it be able to scan it? thanks

 

[redhat27]

You have a valid point there. Ideally the effect should be in a common place: Should affect traffic that is generated outside (like the INPUT chain in the filter table) and also for traffic that is generated inside the LAN (like the FORWARD chain on the filter table)

A common place is the PREROUTING chain of the raw table. I've actually implemented that idea from @Adamm suggestion, and that is indeed working out quite well in my local tests. He asserts that "if you block something in the raw table, its equivalent to blocking in both INPUT and FORWARD chains, but with a significant performance gain as the packets require less processing before being dropped."

I am going to make the changes to both this script and the ya-malware-block script to use the raw table.

Also will post the update to the blockstats alias to be able to see the traffic stats in the wiki (right now it only shows the filter table)