Privacy Filter (Another IPSET Script)

[tomsk]

revision 11 is up

Changelog:

• IPv6 improvements
• Cleanup procedure

Cleanup is a bit dangerous if the user doesn't know what he/she is doing when setting the path. If the path isn't unique the cleanup will wipeout everything in the directory

 

[swetoast]

yeah it can be but im hoping that noone is stupid enough to set it in a dir with other stuff thats important, im banking on that users set the filter to its own dir.

 

[swetoast]

@tomsk my plan is to do everything in tmp in the future cause im tired of getting the opt question so it would be safe to create dir in tmp where it wipes everytime, agree with that approach ?

 

[tomsk]

@tomsk my plan is to do everything in tmp in the future cause im tired of getting the opt question so it would be safe to create dir in tmp where it wipes everytime, agree with that approach ?

well temp will only wipe on restart.. but you might want to consider a small modification to your cleanup which would be safer

cleanup () {
find $path -type f -name partialnameforyourtempfiles* -delete
}

And then call all your temp files partialnameforyourtempfilesXXX.XXX

 

[swetoast]

fixed revision 11 in OP did it Tomsk way instead for cleanup

 

[Goobi]

I went ahead and updated the script to use revision 11... I can't recall what revision I was using previously. I am running 380.64 on a 68U. When I attempt to run the new script, I get the following errors:

# ./privacy-filter.sh
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[[name does not existname does not exist]
]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[[name does not existname does not exist]
]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
find: unrecognized: -type
BusyBox v1.20.2 (2016-12-16 12:24:33 EST) multi-call binary.

Usage: find [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

        -follow         Follow symlinks

Actions:
        ! ACT           Invert ACT's success/failure
        ACT1 [-a] ACT2  If ACT1 fails, stop, else do ACT2
        ACT1 -o ACT2    If ACT1 succeeds, stop, else do ACT2
                        Note: -a has higher priority than -o
        -name PATTERN   Match file name (w/o directory name) to PATTERN
        -iname PATTERN  Case insensitive -name
        -mtime DAYS     mtime is greater than (+N), less than (-N),
                        or exactly N days in the past
If none of the following actions is specified, -print is assumed
        -print          Print file name
        -print0         Print file name, NUL terminated
        -exec CMD ARG ; Run CMD with all instances of {} replaced by
                        file name. Fails if CMD exits with nonzero

If I check to see if the ipset got created I only see 1 IP:

# ipset -L privacy-filter_ipv4
Name: privacy-filter_ipv4
Type: hash:ip
Revision: 0
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 8264
References: 1
Members:
104.131.0.69
**************************************************************************

I have included the script in the event I may have missed something.


Any pointers in the right direction would be greatly appreciated. Thanks in advance.

 

[swetoast]

using entware ? think i need to do a TLDR on what to do if it not working that way i dont have to review all the code.

 

[tomsk]

The [name does not exist] entries mean that hostip is looking up the domain and not finding a match. You could try deleting your existing privacy-filter.list in case it has become corrupted and let the script download the list again. For info the entware version of hostip has been updated (hostip - 1.7.0-1 - 1.9.1-1), so you might want to try the newer version. if hostip is not resolving the domains, then it won't create a list of IP addresses to put in an ipset.

For the "find: unrecognized: -type" problem, that may be due to the compiled version of busybox in that firmware... will have to check.

 

[Goobi]

using entware ? think i need to do a TLDR on what to do if it not working that way i dont have to review all the code.

I do not have entware installed.

 

[Goobi]

The [name does not exist] entries mean that hostip is looking up the domain and not finding a match. You could try deleting your existing privacy-filter.list in case it has become corrupted and let the script download the list again. For info the entware version of hostip has been updated (hostip - 1.7.0-1 - 1.9.1-1), so you might want to try the newer version. if hostip is not resolving the domains, then it won't create a list of IP addresses to put in an ipset.

For the "find: unrecognized: -type" problem, that may be due to the compiled version of busybox in that firmware... will have to check.

I am running version 1.6.0 of hostip but not running entware.

I removed the privacy.list file and the script downloaded it again but produced the same errors. I then had a look at the privacy.list file and noted a bunch of ^M at the end of each line. I removed those and the script ran fine with the exception of the busybox error.

Thanks for pointing me in the right direction. @swetoast thanks for the script as well!

 

[visortgw]

I am running version 1.6.0 of hostip but not running entware.

I removed the privacy.list file and the script downloaded it again but produced the same errors. I then had a look at the privacy.list file and noted a bunch of ^M at the end of each line. I removed those and the script ran fine with the exception of the busybox error.

Thanks for pointing me in the right direction. @swetoast thanks for the script as well!

The "^M"s are embedded carriage returns injected by editing on a non-UNIX (e.g., Windows) host. You can use the dos2unix command on a UNIX host to fix the file.

 

[swetoast]

Revision 12

there might be some breakage but feel free to test and see if it works for you

#!/bin/sh
# Author: Toast
# Contributers: Tomsk
# Revision 12

blocklist=/jffs/privacy-filter.list                     # Set your path here
retries=3                                               # Set number of tries here
failover=eth0                                           # Change only if WAN interface is not detected.

# Dont change this value
regexp_v4=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`
local_v4=`echo "!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/"`
regexp_v6=`echo "^(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4})"`
local_v6=`echo "!(^(fc00::)"`
# Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers
   MATCH_SET='--match-set'
   HASH='hash:ip'
   SYNTAX='add'
   SWAPPED='swap'
   DESTROYED='destroy'
   INET6='family inet6'
   ipsetv=6
    lsmod | grep "xt_set" > /dev/null 2>&1 || \
    for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
    do
         insmod $module
    done
;;
*v4) # Value for Mips Routers
   MATCH_SET='--set'
   HASH='iphash'
   SYNTAX='-q -A'
   SWAPPED='-W'
   DESTROYED='--destroy'
   IPV6=''
    ipsetv=4
    lsmod | grep "ipt_set" > /dev/null 2>&1 || \
    for module in ip_set ip_set_nethash ip_set_iphash ipt_set
    do
         insmod $module
    done
;;
esac

check_online () {
if [ -z "$(which nvram)" ]; then
iface=`grep "$failover" /proc/net/dev`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
else iface=`nvram get wan0_ifname`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
fi }

get_list () {
url=https://gitlab.com/swe_toast/privacy-filter/raw/master/privacy-filter.list
if [ ! -f $blocklist ]
then wget -q --tries=$retries --show-progress $url -O $blocklist; fi }

run_ipv4_block () {
if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
    if [ -z "$(which hostip)" ]; then
        if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                 cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
else    if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
        fi
        
    if [ -f /tmp/privacy-filter_ipv4_presort.part ]; then
        awk $local_v4 /tmp/privacy-filter_ipv4_presort.part > /tmp/privacy-filter_ipv4.prelist; fi
        if [ -f /tmp/privacy-filter_ipv4.prelist ]; then sort -u /tmp/privacy-filter_ipv4.prelist > /tmp/privacy-filter_ipv4_sorted.part; fi
}
        
run_ipv6_block () {
if [ -f /tmp/privacy-filter_ipv6_sorted.part ]; then rm /tmp/privacy-filter_ipv6_sorted.part; fi
    if [ -z "$(which hostip)" ]; then
        if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""; fi
                 cat /tmp/privacy-filter_ipv6_raw.part | grep -oE "$regexp_v6" >> /tmp/privacy-filter_ipv6_presort.part
else    if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""; fi
        fi
        
    if [ -f /tmp/privacy-filter_ipv6_presort.part ]; then
        awk $local_v6 /tmp/privacy-filter_ipv6_presort.part > /tmp/privacy-filter_ipv6.prelist; fi
        if [ -f /tmp/privacy-filter_ipv6.prelist ]; then sort -u /tmp/privacy-filter_ipv6.prelist > /tmp/privacy-filter_ipv6_sorted.part; fi
}
        
run_ipset_4 () {
ipset -L privacy-filter_ipv4 >/dev/null 2>&1
if [ $? -ne 0 ]; then
   if [ "$(ipset --swap privacy-filter_ipv4 privacy-filter_ipv4 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
   nice ipset -N privacy-filter_ipv4 $HASH
   cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv4 {}
fi
else
   nice -n 2 ipset -N privacy-update_ipv4 $HASH
   cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv4 {}
   nice -n 2 ipset $SWAPPED privacy-update_ipv4 privacy-filter_ipv4
   nice -n 2 ipset $DESTROYED privacy-update_ipv4
fi
iptables -L | grep privacy-filter_ipv4 > /dev/null 2>&1
if [ $? -ne 0 ]; then
   nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
else
   nice -n 2 iptables -D FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
   nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
fi }

run_ipset_6 () {
ipset -L privacy-filter_ipv6 >/dev/null 2>&1
if [ $? -ne 0 ]; then
   if [ "$(ipset --swap privacy-filter_ipv6 privacy-filter_ipv6 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
   nice ipset -N privacy-filter_ipv6 $HASH $INET6
   cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv6 {}
fi
else
   nice -n 2 ipset -N privacy-update_ipv6 $HASH $INET6
   cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv6 {}
   nice -n 2 ipset $SWAPPED privacy-update_ipv6 privacy-filter_ipv6
   nice -n 2 ipset $DESTROYED privacy-update_ipv6
fi
iptables -L | grep privacy-filter_ipv6 > /dev/null 2>&1
if [ $? -ne 0 ]; then
   nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
else
   nice -n 2 ip6tables -D FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
   nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
fi }

run_blocklists () {
run_ipv4_block
case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipv6_block; fi ;;
esac }

run_ipset () {
run_ipset_4
case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipset_6; fi  ;;
esac }

cleanup () {
find /tmp -type f -name 'privacy-filter_ipv*.part' -delete
}

check_online
run_blocklists
run_ipset
cleanup

exit $?

 

[wallyg8r]

Revision 12 is working ok for me as was Revision 11.

 

[swetoast]

Thinking of adding dos2unix to the script since i noticed that people copy pastes the list in windows then uploadings it via SFTP thus giving the list windows endings instead of unix endings a simple command can fix that and hopefully save me of a headache so thats something thats upcoming in the next version, anyone else got any suggestions ?

 

[lesandie]

Hi,

I'm in the
374.43_2-22E4j9527 fork release.

Still having problems with the entware hostip (1.9.1) and [name does not exist] output.

I'll try to debug and change the code if i find what's wrong.

Nice work guys!

 

[swetoast]

Name does not exist can happen sometimes i you have adblocker or the domain doesnt have an ip, no need for debuging there is nothing wrong.

 

[john9527]

Hi,

I'm in the
374.43_2-22E4j9527 fork release.

Still having problems with the entware hostip (1.9.1) and [name does not exist] output.

I'll try to debug and change the code if i find what's wrong.

Nice work guys!

You may be running into a conflict.....since I include native dnscrypt support, hostip is also installed as part of the base firmware (in /usr/sbin).

 

[swetoast]

But @john9527 the dns server should still resolve the ip that should still work as intended on your fork right ?

 

[john9527]

But @john9527 the dns server should still resolve the ip that should still work as intended on your fork right ?

Yes, unless the hostip that's included with dnscrypt has a different syntax from the entware version (probably not) or there is a mismatch somehow in any linked libraries.

 

[swetoast]

mind just giving me a print out of hostip from your distro ?

hostip --help