iblocklist.com generic ipset loader for ipset v6 and v4

[Xentrk]

Would the correct IPs for each location be added to the WhitelistDomains ipset? I mean if you just whitelist:
yahoo.com
mail.yahoo.com
mg.mail.yahoo.com
would that not be enough? Even if mg.mail.yahoo.com resolves to different IPs in different locations, it would still whitelist the IPs it resolves to at each location and for the traffic each location handles.

I will give it a try. I did not whitelist mail.yahoo.com but will now.

Also, here is another url in the blacklist-domains.txt that reports as invalid:

nslookup: can't resolve 'public-family.api.account.microsoft.com'

 

[Xentrk]

I found an issue with iblocklist-loader creating a problem with the ip address assigned to pixelserv-tls. I have been having issues the past two weeks with AB-Solution stating something is wrong with my pixelserv-tls. I believe it is because the ip address assigned to pixelserv-tls is in the BlackListDomains file. Not sure what URL is doing this. Since the white list file are URL's, how can I white list the pixelserv IP address? Thanks.

admin@RT-AC88U:/jffs/scripts# MatchIP 192.168.3.2
192.168.3.2 not found in YAMalwareBlockCIDR
192.168.3.2 not found in YAMalwareBlock2IP
192.168.3.2 not found in YAMalwareBlock1IP
192.168.3.2 not found in WhitelistDomains
192.168.3.2 found in BlacklistDomains
192.168.3.2 not found in BluetackProxyCIDR
192.168.3.2 not found in BluetackProxySingle
192.168.3.2 not found in BluetackWebexploitCIDR
192.168.3.2 not found in BluetackWebexploitSingle
192.168.3.2 not found in BluetackDshieldCIDR
192.168.3.2 not found in BluetackDshieldSingle
192.168.3.2 not found in BlockedCountries
192.168.3.2 not found in TorNodes
192.168.3.2 found in Whitelist
192.168.3.2 not found in Blacklist
192.168.3.2 not found in MicrosoftSpyServers
192.168.3.2 not found in WhitelistSRCPort
192.168.3.2 found in Whitelist
192.168.3.2 not found in Blacklist

 

[redhat27]

Sorry for the late reply @Xentrk work was keeping me busy. Will likely keep me occupied in the near future too

here is another url in the blacklist-domains.txt that reports as invalid

Removed the invalid entry. Thanks for reporting.

I found an issue with iblocklist-loader creating a problem with the ip address assigned to pixelserv-tls
:
I believe it is because the ip address assigned to pixelserv-tls is in the BlackListDomains file

Whatever domain is resolving to the local IP, is now being blocked on the firewall instead of in the dns lookup. I fail to see why that would be an issue. I have several in the blacklist-domains.txt file that resolve to a local IP, but that does not pose to be an issue for me. You can manually delete that IP from BlacklistDomains with:

ipset del BlacklistDomains 192.168.3.2

then re-check with the shell function to find IP in ipsets.
It would then be blocked on the dns lookup

 

[ScottW]

Looks like I need some help...

When I run iblocklist-loader.sh, it appears to work fine. No errors, list files are created, whitelist and blacklist are processed.

But when I look at iptables, I don't see any blocklist rules. I expected to see some "-m set --match" entries in the FORWARD chain. What am I doing wrong?

admin@RT-AC88U-0000:/tmp/mnt/ASUS/ipset# sh iblocklist-loader.sh
iblocklist-loader.sh: Started processing BluetackDshield blocklist
iblocklist-loader.sh: Loaded BluetackDshieldSingle blocklist with 20 entries
iblocklist-loader.sh: Loaded BluetackDshieldCIDR blocklist with 20 entries
iblocklist-loader.sh: Started processing BluetackWebexploit blocklist
iblocklist-loader.sh: Loaded BluetackWebexploitSingle blocklist with 1458 entries
iblocklist-loader.sh: Loaded BluetackWebexploitCIDR blocklist with 38 entries
iblocklist-loader.sh: Added BlacklistDomains (1 entries)
iblocklist-loader.sh: Added WhitelistDomains (19 entries)

admin@RT-AC88U-0000:/tmp/mnt/ASUS/ipset# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N FUPNP
-N NSFW
-N PControls
-N SECURITY
-N SECURITY_PROTECT
-N iptfromlan
-N ipttolan
-N logaccept
-N logdrop
-A INPUT -i tun21 -j ACCEPT
-A INPUT -p udp -m udp --dport 1194 -j ACCEPT
-A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j DROP
-A INPUT -p tcp -m multiport --dports 22 -j SECURITY_PROTECT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m state --state INVALID -j DROP
-A INPUT -i br0 -m state --state NEW -j ACCEPT
-A INPUT -i lo -m state --state NEW -j ACCEPT
-A INPUT -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A INPUT -p icmp -m icmp ! --icmp-type 8 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -o br0 -j ipttolan
-A FORWARD -i br0 -j iptfromlan
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun21 -j ACCEPT
-A FORWARD ! -i br0 -o eth0 -j DROP
-A FORWARD -i eth0 -m state --state INVALID -j DROP
-A FORWARD -i br0 -o br0 -j ACCEPT
-A FORWARD -j NSFW
-A FORWARD -m conntrack --ctstate DNAT -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
-A PControls -j ACCEPT
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j RETURN
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -m limit --limit 1/sec -j RETURN
-A SECURITY -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK RST -j DROP
-A SECURITY -p icmp -m icmp --icmp-type 8 -m limit --limit 1/sec -j RETURN
-A SECURITY -p icmp -m icmp --icmp-type 8 -j DROP
-A SECURITY -j RETURN
-A iptfromlan -o eth0 -m account--aaddr 192.168.1.0/255.255.255.0 --aname lan  -j RETURN
-A iptfromlan -o tun21 -m account--aaddr 192.168.1.0/255.255.255.0 --aname lan  -j RETURN
-A ipttolan -i eth0 -m account--aaddr 192.168.1.0/255.255.255.0 --aname lan  -j RETURN
-A ipttolan -i tun21 -m account--aaddr 192.168.1.0/255.255.255.0 --aname lan  -j RETURN
-A logaccept -m state --state NEW -j LOG --log-prefix "ACCEPT " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logaccept -j ACCEPT
-A logdrop -m state --state NEW -j LOG --log-prefix "DROP " --log-tcp-sequence --log-tcp-options --log-ip-options
-A logdrop -j DROP
admin@RT-AC88U-0000:/tmp/mnt/ASUS/ipset#

 

[redhat27]

@ScottW its in the raw table, PREROUTING.
See post #186

 

[ScottW]

@ScottW its in the raw table, PREROUTING.
See post #186

Thanks! Saw some references to prerouting but didn't connect... "iptables -t raw -S" (or "iptables -vL -t raw") correctly shows all the blocking sets in prerouting.

Thanks for the help, and thanks for the script!

 

[Xentrk]

Sorry for the late reply @Xentrk work was keeping me busy. Will likely keep me occupied in the near future too

I know that feeling all to well. My last project was managing a data center move for state agencies into a new State data center. Of course, they need to move during off hours, holidays and weekends. The boss thought I should not be compensated for all of the extra time since I was salary exempt. My good work was rewarded by being assigned other projects that also required even more off hour work in addition to a normal work week without being offered any comp time. That is when I said, F.I, I'm moving to Thailand!

You can manually delete that IP from BlacklistDomains with:

ipset del BlacklistDomains 192.168.3.2

then re-check with the shell function to find IP in ipsets.
It would then be blocked on the dns lookup

The ipset del BlacklistDomains command fixed the pixelserv-tls issue for me AB-Solution was reporting. This has been a mystery the past few weeks. I first reported the issue in the AB-Solution thread several weeks ago. My fix has been to remove ab-solution and entware and reinstall. It reports everything good until I run the scripts in services-start. AB-Solution reported pixelserv could not start or could not get a reply from the ip address. Yet, when I did a "ps | grep pixelserv", I saw the two process ids for pixelserv assisgned to admin and nobody. I was not able to narrow it down until just now. The MatchIP utility your wrote has been so helpful in finding out what is blocking certain sites and in finding the root cause of this problem. Letting the @thelonelycoder know in case this pops up in his thread.

I found out this was happening on all three routers. Each one has a different IP address. But they all start with 192.168

 

[thelonelycoder]

update-hosts.add version 3.8.2 is now available for AB-Solution

Use 12 or cu in the AB UI to update to this latest addon version.

Changelog update-hosts.add v3.8.2:
- adds auto-whitelist support for @redhat27's iblocklist-loader.sh and ya-malware-block scripts.sh.
- auto-whitelists domains used in the above scripts to use full potential of them

Note that the auto-whitelisting of domains only works if the above scripts are installed in the default locations:
/jffs/scripts/iblocklist-loader.sh and/or ya-malware-block.sh
/jffs/ipset_lists/blacklist-domains.txt and/or ya-malware-block.urls

@Adamm's and @Martineau's scripts do not need whitelisting in AB.

 

[shooter40sw]

update-hosts.add version 3.8.2 is now available for AB-Solution

Use 22 or cu in the AB UI to update to this latest addon version.

Changelog update-hosts.add v3.8.2:
- adds auto-whitelist support for @redhat27's iblocklist-loader.sh and ya-malware-block scripts.sh.
- auto-whitelists domains used in the above scripts to use full potential of them

Note that the auto-whitelisting of domains only works if the above scripts are installed in the default locations:
/jffs/scripts/iblocklist-loader.sh and/or ya-malware-block.sh
/jffs/ipset_lists/blacklist-domains.txt and/or ya-malware-block.urls

@Adamm's and @Martineau's scripts do not need whitelisting in AB.

Hi @thelonelycoder , is there any way I can edit it to adapt the locations to my install? I use both scripts but not on default location.
Thanks

 

[thelonelycoder]

Hi @thelonelycoder , is there any way I can edit it to adapt the locations to my install? I use both scripts but not on default location.
Thanks

The lines to edit in update-hosts.add are 182 and 185 for iblocklist-loader.sh:

if [ -s /jffs/ipset_lists/blacklist-domains.txt ] && [ -s /jffs/scripts/iblocklist-loader.sh ];then

and

grep "^[^#;]" /jffs/ipset_lists/blacklist-domains.txt | awk '{print $1}' | sed -e 's/^/ /; s/$/$/' >> $whitelist.tmp

And 188 and 191 for ya-malware-block.sh:

if [ -s /jffs/ipset_lists/ya-malware-block.urls ] && [ -s /jffs/scripts/ya-malware-block.sh ];then

and

grep "^[^#;]" /jffs/ipset_lists/ya-malware-block.urls | sed 's/http[s]*:\/\///;s|\/.*||' | sed -e 's/^/ /; s/$/$/' >> $whitelist.tmp

Both files in the first line need to be present for it to kick in.

I did not use a search function to find the files because some may have backup files saved somewhere in the file system and then I would have no idea which one to use.

Note that AB may silently replace the file in a future update or for other reasons only the coder knows...

 

[redhat27]

@thelonelycoder I do not think any special handling is needed for
[ -s /jffs/ipset_lists/ya-malware-block.urls ] && [ -s /jffs/scripts/ya-malware-block.sh]

The urls listed there are just from raw.githubusercontent.com

@Adamm's and @Martineau's scripts do not need whitelisting in AB.

Not sure about Martineau, but Adamm's script also has this domain (and a couple more) for his script sources.

I do not think ya-malware-block.sh needs any whitelisting on AB side.

 

[thelonelycoder]

@thelonelycoder I do not think any special handling is needed for
[ -s /jffs/ipset_lists/ya-malware-block.urls ] && [ -s /jffs/scripts/ya-malware-block.sh]

The urls listed there are just from raw.githubusercontent.com


Not sure about Martineau, but Adamm's script also has this domain (and a couple more) for his script sources.

I do not think ya-malware-block.sh needs any whitelisting on AB side.

I just include it to be on the safe side, AB auto-adds all used hosts file domains, might as well add the github (or whatever domain users add to that file) in case it gets added for a reason by one of the hosts file providers.

 

[joesixpack]

@joesixpack Some of your concerns are valid. This script specifically caters to loading lists form the iblocklist.com site, with some limited customizations. For general overall security, I would recommend you to try out the ya-malware-block script that blocks a wide variety of malware sources. Please give post #1 a good read. It is quite fast and does not leave your LAN exposed while it is running (for that matter neither does this one). Let me know if that script would work for you. For myself, I've enabled Level4 on my router, but note it is disabled by default (Levels 1 to 3 enabled)

Thanks, I've installed it into services-start. That alleviates most of my concern.

If you plan to keep using this, I could have a WhitelistCIDR set created if (lets say) a /jffs/ipset_lists/whitelist-cidr.txt exists with CIDR entries. LMK if that would work.

That would be a good idea so we can whitelist the cloud CIDRs.

So far I've replicated all of the non-stale iBlocklists available into PeerBlock and additionally disabled the port 80/443 passthru. Consequently, I've had to disable Malicious, Piracy and Porn for being overbroad or nonsensical. After that, I've only had to whitelist four IP's, so this is very good indeed. I'll give it a week to see what else gets blocked before I load them into the router.

The target for a match is either DROP/REJECT (choice) or ACCEPT. I've not opted to use logdrop or logaccept as I do not like too much chatter in the syslog. If you have firewall logging enabled, just change this line, for example to "logdrop" instead of "DROP"

I will try this and see if it can replace PeerBlock's logging.

Okay, I see your point there. I can make the processing order like this:

WHITELIST_DOMAINS_FILE
BLACKLIST_DOMAINS_FILE
BLOCKLIST_INDEXES
ALLOWLIST_INDEXES

or, possibly

WHITELIST_DOMAINS_FILE
WHITELIST_CIDR_FILE (not yet there)
BLACKLIST_DOMAINS_FILE
BLACKLIST_CIDR_FILE (not yet there)
BLOCKLIST_INDEXES
ALLOWLIST_INDEXES

Edit: Oh, welcome to the snbforums, BTW

What is the processing order at present? I think maybe the issue here is that any allow or block stops further processing. So if, say, Amazon EC2 CIDR range is whitelisted but there's an infected IP on in it, it won't be blocked.

 

[redhat27]

What is the processing order at present?

This

That would be a good idea so we can whitelist the cloud CIDRs.

I will implement a custom WhitelistCIDR and BlacklistCIDR soon (when time permits). Thanks for the suggestion.

 

[Xentrk]

Hi @redhat27,
Are you watching the NBA Finals tonight?

I ran a manual update last night and noticed there are some entries in blacklist-domains.txt that appear to no longer be valid:

watson.telemetry.microsoft.com.nsatc.net # United States, WA, Redmond
oca.telemetry.microsoft.com.nsatc.net # United States, WA, Redmond
telecommand.telemetry.microsoft.com.nsatc.net # United States, WA, Redmond

 

[Xentrk]

Also, on the GitHub site, it lists iblocklist-loader.sh and iblocklist-loader-v2.sh. In the comment section, both are listed as being version 1.1, which causes some confusion. Thanks for your support.

 

[redhat27]

Are you watching the NBA Finals tonight?

I'll catch the next one on the 4th. The Warriors won yesterday.

I ran a manual update last night and noticed there are some entries in blacklist-domains.txt that appear to no longer be valid:

Please feel free to contribute to my github repo. Just raise a pull request and I'll merge. If you're not sure how, let me know, and I'll make the changes.

In the comment section, both are listed as being version 1.1

I will make changes to the v2 (extended version of the original script) so that it will have a CIDR whitelist and blacklist (in addition to the domain whitelist/blacklist) It will then be version 1.2, while the original will remain at 1.1
I know the original and v2 can cause confusion. I've thought about it myself. Do you think it would be better to have those renamed to original and extended? The first one will be suited for most use cases, but there are a lot more options and configurability on the v2

 

[Xentrk]

I'll catch the next one on the 4th. The Warriors won yesterday.

I must be watching the replay then. That explains why it's on ESPN3. I just started coffee number 3 and still shaking off the morning dew..

I will make changes to the v2 (extended version of the original script) so that it will have a CIDR whitelist and blacklist (in addition to the domain whitelist/blacklist) It will then be version 1.2, while the original will remain at 1.1
I know the original and v2 can cause confusion. I've thought about it myself. Do you think it would be better to have those renamed to original and extended? The first one will be suited for most use cases, but there are a lot more options and configurability on the v2

I was wondering the same thing. I think your idea of renaming original and extended as it will help differentiate the functionality difference between the two scripts. Another idea is to use something like iblocklist-loader.sh (original) and iblocklist-loader.sh (extended) for the file name. Not sure if this is a valid file name format for GitHub though and if their checks will allow for it.

 

[redhat27]

I must be watching the replay then

Oops... Sorry for the spoiler.

 

[Xentrk]

Oops... Sorry for the spoiler.

No problem. The win made the news yesterday so I was aware of it. I was just fuzzed brain on which game as I had not been following the series too closely. I lived in the metro Detroit area for many years and followed the Pistons. I knew you lived on the left coast and assumed you had some interest in the series as a result. Okay, back to iblocklist-loader.sh!!