What blocking scripts to install?

[Adamm]

Adamm, while you are at it, in would be nice to have a link in root, the way ab-s does. So when I putty in I can do "sh firewall etc,etc" instead of sh /jffs/scripts/firewall etc etc.

I've added it myself.

Skynet adds a symbolic link to /opt/bin to do just this if it detects Entware installed.

 

[thelonelycoder]

Skynet adds a symbolic link to /opt/bin to do just this if it detects Entware installed.

Same with AB-Solution:
Type ab and hit the TAB key:
ab-solution

 

[thelonelycoder]

@Adamm and other script writers:
I include the users whitelist in the /jffs/shared-AB-whitelist, so you may remove code that includes the /adblocking/whitelist.txt as soon as AB 3.8.2 is released. All used hosts file domains by AB are by default included in that list. It is sorted for easier reading.

In return I have built in a routine to include all /jffs/shared-*-whitelist files to whitelist the included domains in AB.

 

[thelonelycoder]

@Adamm could you add an echo before and after the shared list is run, this way users know it is being worked on.
I have a few obsolete entries in the list that I would like to keep in but they cannot be resolved.
Skynet then outputs some nslookup no resolve lines for these. Would be helpful for the user to know where this comes from.

 

[Adamm]

@Adamm could you add an echo before and after the shared list is run, this way users know it is being worked on.
I have a few obsolete entries in the list that I would like to keep in but they cannot be resolved.
Skynet then outputs some nslookup no resolve lines for these. Would be helpful for the user to know where this comes from.

I added an echo and silenced these errors as its more up to the other scriptwriters to correct/remove these non resolving domains.

 

[Adamm]

@thelonelycoder I ended up improving this solution to support any shared whitelist file detected, [] doesn't support wildcards, so I had to use the find command to get desired functionality.

        if [ -n "$(find /jffs -maxdepth 1 -name 'shared-*-whitelist')" ]; then
            echo "Whitelisting Shared Domains"
            grep -hvF "#" /jffs/shared-*-whitelist | cut -d '/' -f3 | awk '!x[$0]++' | while IFS= read -r domain; do
                for ip in $(Domain_Lookup "$domain" 2> /dev/null); do
                    ipset -q -A Whitelist "$ip"
                    ipset -q -D Blacklist "$ip"
                done
            done
        fi

One of the issues I ran into was when resolving hostnames with nslookup, it doesn't support input with backslashes. So I use the cut command to work around this. The only time it will fail though is if a URL doesn't contain http:// but only contains a extended path e.g;

www.google.com/testpage

But any of the following would work

https://www.google.com
https://www.google.com/testpage
www.google.com

Hopefully any URL's in your whitelist are compatible with this (and anyone who decides to use this in future). This can probably be done better by specifically cutting https:// and http:// then any text trailing if there is a remaining /, but this is a temporary solution for now until I find something more elegant without spawning a bunch of different utilities.

 

[thelonelycoder]

@thelonelycoder I ended up improving this solution to support any shared whitelist file detected, [] doesn't support wildcards, so I had to use the find command to get desired functionality.

        if [ -n "$(find /jffs -maxdepth 1 -name 'shared-*-whitelist')" ]; then
            echo "Whitelisting Shared Domains"
            grep -hvF "#" /jffs/shared-*-whitelist | cut -d '/' -f3 | awk '!x[$0]++' | while IFS= read -r domain; do
                for ip in $(Domain_Lookup "$domain" 2> /dev/null); do
                    ipset -q -A Whitelist "$ip"
                    ipset -q -D Blacklist "$ip"
                done
            done
        fi

One of the issues I ran into was when resolving hostnames with nslookup, it doesn't support input with backslashes. So I use the cut command to work around this. The only time it will fail though is if a URL doesn't contain http:// but only contains a extended path e.g;

www.google.com/testpage

But any of the following would work

https://www.google.com
https://www.google.com/testpage
www.google.com

Hopefully any URL's in your whitelist are compatible with this (and anyone who decides to use this in future). This can probably be done better by specifically cutting https:// and http:// then any text trailing if there is a remaining /, but this is a temporary solution for now until I find something more elegant without spawning a bunch of different utilities.

I strip everything, leaving only www.domain.com or domain.com.
I use this to include them all.
The extra space indention and line end ($) helps me to target the exact string. Both are not done for the shared list.

for wlists in /jffs/shared-*-whitelist;do

if [ -e "$wlists" ] && [ "$wlists" != "/jffs/shared-AB-whitelist" ]; then
echo -e " additionally including $wlists\n to whitelist this scripts domains\n"
# indent and add $ to end of line, append to whitelist.tmp
grep "^[^#;]" $wlists | sed -e 's/^/ /; s/$/$/' >> $whitelist.tmp
# sort and remove duplicates in new temp whitelist
sort -u $whitelist.tmp -o $whitelist.tmp
fi

done

 

[Adamm]

I strip everything, leaving only www.domain.com or domain.com.
I use this to include them all.
The extra space indention and line end ($) helps me to target the exact string. Both are not done for the shared list.

for wlists in /jffs/shared-*-whitelist;do

if [ -e "$wlists" ] && [ "$wlists" != "/jffs/shared-AB-whitelist" ]; then
echo -e " additionally including $wlists\n to whitelist this scripts domains\n"
# indent and add $ to end of line, append to whitelist.tmp
grep "^[^#;]" $wlists | sed -e 's/^/ /; s/$/$/' >> $whitelist.tmp
# sort and remove duplicates in new temp whitelist
sort -u $whitelist.tmp -o $whitelist.tmp
fi

done

Thanks, it started to bug me after I posted this so I settled on the following.

grep -hvF "#" /jffs/shared-*-whitelist | sed 's~https://~~g; s~http://~~g' | cut -d '/' -f1 | awk '!x[$0]++'

 

[thelonelycoder]

Thanks, it started to bug me after I posted this so I settled on the following.

grep -hvF "#" /jffs/shared-*-whitelist | sed 's~https://~~g; s~http://~~g' | cut -d '/' -f1 | awk '!x[$0]++'

This is how I strip url's for whitelists and blacklists, shared or not, leaving the bare domain, with www if part of url:

sed 's/http[s]*:\/\///;s|\/.*||'

 

[Adamm]

This is how I strip url's for whitelists and blacklists, shared or not, leaving the bare domain, with www if part of url:

sed 's/http[s]*:\/\///;s|\/.*||'

Perfect, exactly what I had in mind, sed regex gives me headaches so couldn't get it exactly how I wanted. Thanks

 

[thelonelycoder]

Perfect, exactly what I had in mind, sed regex gives me headaches so couldn't get it exactly how I wanted. Thanks

This is two sed's in one, neat and clean.

 

[redhat27]

Pinging @redhat27, @Martineau to read above for whitelisting domains.
Thanks.

Just saw this... Will update soon as I get a chance. Sorry for being so infrequent here these days.

 

[redhat27]

Skynet (Block. from inside or outside or both?)
iblocklist-loader (Block. from inside or outside or both?)
AB_Solution (Ad blocking from inside)
ya-malware-block (Ad blocking from inside)

Sorry for the late reply.

AB solution is an excellent script for advertisement blocking, with different pre-packaged known sources for advertisements that you can choose to block. The blocking is on the DNS lookup, it will redirect the DNS lookup locally (to pixelserv if you have that installed, or to 0.0.0.0 on the router)

iblocklist-loader is very configurable and can block on inbound and/or outbound traffic. It is tailored to use the public blocklists published on the iblocklist site

ya-malware-block aims to stop access from any identified malware site and uses the firehol blocking lists. It also blocks on both inbound and outbound, except the initial SYN request (block on the ACK) So your LAN machines are actually also blocked from being able to make outbound connections to malware sites.

Adamm maintains the skynet script and also has various options and configurability.

AB-Solution, ya-malware-block and iblocklist-loader do not conflict with each other. I use all three. AB-Solution is DNS poisoning while ya-malware-block, iblocklist-loader, skynet are firewall.

The ya-malware-block and iblocklist-loader can co-exist: They have different purposes, although there may be some overlap with some of the lists. If an IP is blocked on more than one lists from the two or more scripts, there is no harm: The iptables filter that blocks it first wins, the other rule will not be enacted.