Menu
What's new
By:
Filters Better Search

Privacy Filter (Another IPSET Script)

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

swetoast said:
revision 11 is up

Changelog:
  • IPv6 improvements
  • Cleanup procedure
Click to expand...
Cleanup is a bit dangerous if the user doesn't know what he/she is doing when setting the path. If the path isn't unique the cleanup will wipeout everything in the directory
 
yeah it can be but im hoping that noone is stupid enough to set it in a dir with other stuff thats important, im banking on that users set the filter to its own dir.
 
@tomsk my plan is to do everything in tmp in the future cause im tired of getting the opt question so it would be safe to create dir in tmp where it wipes everytime, agree with that approach ?
 
swetoast said:
@tomsk my plan is to do everything in tmp in the future cause im tired of getting the opt question so it would be safe to create dir in tmp where it wipes everytime, agree with that approach ?
Click to expand...
well temp will only wipe on restart.. but you might want to consider a small modification to your cleanup which would be safer
Code: 
cleanup () {
find $path -type f -name partialnameforyourtempfiles* -delete
}

And then call all your temp files partialnameforyourtempfilesXXX.XXX
 
Last edited:
I went ahead and updated the script to use revision 11... I can't recall what revision I was using previously. I am running 380.64 on a 68U. When I attempt to run the new script, I get the following errors:

Code: 
# ./privacy-filter.sh
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[[name does not existname does not exist]
]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[[name does not existname does not exist]
]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
[name does not exist]
find: unrecognized: -type
BusyBox v1.20.2 (2016-12-16 12:24:33 EST) multi-call binary.

Usage: find [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

        -follow         Follow symlinks

Actions:
        ! ACT           Invert ACT's success/failure
        ACT1 [-a] ACT2  If ACT1 fails, stop, else do ACT2
        ACT1 -o ACT2    If ACT1 succeeds, stop, else do ACT2
                        Note: -a has higher priority than -o
        -name PATTERN   Match file name (w/o directory name) to PATTERN
        -iname PATTERN  Case insensitive -name
        -mtime DAYS     mtime is greater than (+N), less than (-N),
                        or exactly N days in the past
If none of the following actions is specified, -print is assumed
        -print          Print file name
        -print0         Print file name, NUL terminated
        -exec CMD ARG ; Run CMD with all instances of {} replaced by
                        file name. Fails if CMD exits with nonzero

If I check to see if the ipset got created I only see 1 IP:

# ipset -L privacy-filter_ipv4
Name: privacy-filter_ipv4
Type: hash:ip
Revision: 0
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 8264
References: 1
Members:
104.131.0.69
**************************************************************************

I have included the script in the event I may have missed something.


Any pointers in the right direction would be greatly appreciated. Thanks in advance.
 

Attachments

  • privacy-filter.sh.txt
    6.4 KB · Views: 369
using entware ? think i need to do a TLDR on what to do if it not working that way i dont have to review all the code.
 
The [name does not exist] entries mean that hostip is looking up the domain and not finding a match. You could try deleting your existing privacy-filter.list in case it has become corrupted and let the script download the list again. For info the entware version of hostip has been updated (hostip - 1.7.0-1 - 1.9.1-1), so you might want to try the newer version. if hostip is not resolving the domains, then it won't create a list of IP addresses to put in an ipset.

For the "find: unrecognized: -type" problem, that may be due to the compiled version of busybox in that firmware... will have to check.
 
Last edited:
tomsk said:
The [name does not exist] entries mean that hostip is looking up the domain and not finding a match. You could try deleting your existing privacy-filter.list in case it has become corrupted and let the script download the list again. For info the entware version of hostip has been updated (hostip - 1.7.0-1 - 1.9.1-1), so you might want to try the newer version. if hostip is not resolving the domains, then it won't create a list of IP addresses to put in an ipset.

For the "find: unrecognized: -type" problem, that may be due to the compiled version of busybox in that firmware... will have to check.
Click to expand...

I am running version 1.6.0 of hostip but not running entware.

I removed the privacy.list file and the script downloaded it again but produced the same errors. I then had a look at the privacy.list file and noted a bunch of ^M at the end of each line. I removed those and the script ran fine with the exception of the busybox error.

Thanks for pointing me in the right direction. @swetoast thanks for the script as well!
 
Goobi said:
I am running version 1.6.0 of hostip but not running entware.

I removed the privacy.list file and the script downloaded it again but produced the same errors. I then had a look at the privacy.list file and noted a bunch of ^M at the end of each line. I removed those and the script ran fine with the exception of the busybox error.

Thanks for pointing me in the right direction. @swetoast thanks for the script as well!
Click to expand...
The "^M"s are embedded carriage returns injected by editing on a non-UNIX (e.g., Windows) host. You can use the dos2unix command on a UNIX host to fix the file.
 
Revision 12

there might be some breakage but feel free to test and see if it works for you :)

Code: 
#!/bin/sh
# Author: Toast
# Contributers: Tomsk
# Revision 12

blocklist=/jffs/privacy-filter.list                     # Set your path here
retries=3                                               # Set number of tries here
failover=eth0                                           # Change only if WAN interface is not detected.

# Dont change this value
regexp_v4=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`
local_v4=`echo "!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/"`
regexp_v6=`echo "^(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4}:)+(:)?(([0-9a-f]){1,4})"`
local_v6=`echo "!(^(fc00::)"`
# Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers
   MATCH_SET='--match-set'
   HASH='hash:ip'
   SYNTAX='add'
   SWAPPED='swap'
   DESTROYED='destroy'
   INET6='family inet6'
   ipsetv=6
    lsmod | grep "xt_set" > /dev/null 2>&1 || \
    for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
    do
         insmod $module
    done
;;
*v4) # Value for Mips Routers
   MATCH_SET='--set'
   HASH='iphash'
   SYNTAX='-q -A'
   SWAPPED='-W'
   DESTROYED='--destroy'
   IPV6=''
    ipsetv=4
    lsmod | grep "ipt_set" > /dev/null 2>&1 || \
    for module in ip_set ip_set_nethash ip_set_iphash ipt_set
    do
         insmod $module
    done
;;
esac

check_online () {
if [ -z "$(which nvram)" ]; then
iface=`grep "$failover" /proc/net/dev`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
else iface=`nvram get wan0_ifname`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
fi }

get_list () {
url=https://gitlab.com/swe_toast/privacy-filter/raw/master/privacy-filter.list
if [ ! -f $blocklist ]
then wget -q --tries=$retries --show-progress $url -O $blocklist; fi }

run_ipv4_block () {
if [ -f /tmp/privacy-filter_ipv4_sorted.part ]; then rm /tmp/privacy-filter_ipv4_sorted.part; fi
    if [ -z "$(which hostip)" ]; then
        if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -4 {} | head -1 >> "/tmp/privacy-filter_ipv4_raw.part""; fi
                 cat /tmp/privacy-filter_ipv4_raw.part | grep -oE "$regexp_v4" >> /tmp/privacy-filter_ipv4_presort.part
else    if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip {} >> "/tmp/privacy-filter_ipv4.prelist""; fi
        fi
        
    if [ -f /tmp/privacy-filter_ipv4_presort.part ]; then
        awk $local_v4 /tmp/privacy-filter_ipv4_presort.part > /tmp/privacy-filter_ipv4.prelist; fi
        if [ -f /tmp/privacy-filter_ipv4.prelist ]; then sort -u /tmp/privacy-filter_ipv4.prelist > /tmp/privacy-filter_ipv4_sorted.part; fi
}
        
run_ipv6_block () {
if [ -f /tmp/privacy-filter_ipv6_sorted.part ]; then rm /tmp/privacy-filter_ipv6_sorted.part; fi
    if [ -z "$(which hostip)" ]; then
        if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "traceroute -6 {} | head -1 >> "/tmp/privacy-filter_ipv6_raw.part""; fi
                 cat /tmp/privacy-filter_ipv6_raw.part | grep -oE "$regexp_v6" >> /tmp/privacy-filter_ipv6_presort.part
else    if [ -z "$(which /opt/bin/xargs)" ]
            then cat $blocklist | xargs -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""
            else cat $blocklist | /opt/bin/xargs -P 10 -n 5 -I {} sh -c "hostip -6 {} >> "/tmp/privacy-filter_ipv6.prelist""; fi
        fi
        
    if [ -f /tmp/privacy-filter_ipv6_presort.part ]; then
        awk $local_v6 /tmp/privacy-filter_ipv6_presort.part > /tmp/privacy-filter_ipv6.prelist; fi
        if [ -f /tmp/privacy-filter_ipv6.prelist ]; then sort -u /tmp/privacy-filter_ipv6.prelist > /tmp/privacy-filter_ipv6_sorted.part; fi
}
        
run_ipset_4 () {
ipset -L privacy-filter_ipv4 >/dev/null 2>&1
if [ $? -ne 0 ]; then
   if [ "$(ipset --swap privacy-filter_ipv4 privacy-filter_ipv4 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
   nice ipset -N privacy-filter_ipv4 $HASH
   cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv4 {}
fi
else
   nice -n 2 ipset -N privacy-update_ipv4 $HASH
   cat /tmp/privacy-filter_ipv4_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv4 {}
   nice -n 2 ipset $SWAPPED privacy-update_ipv4 privacy-filter_ipv4
   nice -n 2 ipset $DESTROYED privacy-update_ipv4
fi
iptables -L | grep privacy-filter_ipv4 > /dev/null 2>&1
if [ $? -ne 0 ]; then
   nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
else
   nice -n 2 iptables -D FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
   nice -n 2 iptables -I FORWARD -m set $MATCH_SET privacy-filter_ipv4 src,dst -j REJECT
fi }

run_ipset_6 () {
ipset -L privacy-filter_ipv6 >/dev/null 2>&1
if [ $? -ne 0 ]; then
   if [ "$(ipset --swap privacy-filter_ipv6 privacy-filter_ipv6 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
   nice ipset -N privacy-filter_ipv6 $HASH $INET6
   cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-filter_ipv6 {}
fi
else
   nice -n 2 ipset -N privacy-update_ipv6 $HASH $INET6
   cat /tmp/privacy-filter_ipv6_sorted.part | xargs -I {} ipset $SYNTAX privacy-update_ipv6 {}
   nice -n 2 ipset $SWAPPED privacy-update_ipv6 privacy-filter_ipv6
   nice -n 2 ipset $DESTROYED privacy-update_ipv6
fi
iptables -L | grep privacy-filter_ipv6 > /dev/null 2>&1
if [ $? -ne 0 ]; then
   nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
else
   nice -n 2 ip6tables -D FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
   nice -n 2 ip6tables -I FORWARD -m set $MATCH_SET privacy-filter_ipv6 src,dst -j REJECT
fi }

run_blocklists () {
run_ipv4_block
case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipv6_block; fi ;;
esac }

run_ipset () {
run_ipset_4
case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) if [ "$(cat /proc/net/if_inet6 | wc -l)" -gt "0" ]; then run_ipset_6; fi  ;;
esac }

cleanup () {
find /tmp -type f -name 'privacy-filter_ipv*.part' -delete
}

check_online
run_blocklists
run_ipset
cleanup

exit $?
 
Thinking of adding dos2unix to the script since i noticed that people copy pastes the list in windows then uploadings it via SFTP thus giving the list windows endings instead of unix endings a simple command can fix that and hopefully save me of a headache so thats something thats upcoming in the next version, anyone else got any suggestions ?
 
Hi,

I'm in the
374.43_2-22E4j9527 fork release.

Still having problems with the entware hostip (1.9.1) and [name does not exist] output.

I'll try to debug and change the code if i find what's wrong.

Nice work guys! :)
 
Name does not exist can happen sometimes i you have adblocker or the domain doesnt have an ip, no need for debuging there is nothing wrong.
 
lesandie said:
Hi,

I'm in the
374.43_2-22E4j9527 fork release.

Still having problems with the entware hostip (1.9.1) and [name does not exist] output.

I'll try to debug and change the code if i find what's wrong.

Nice work guys! :)
Click to expand...
You may be running into a conflict.....since I include native dnscrypt support, hostip is also installed as part of the base firmware (in /usr/sbin).
 
swetoast said:
But @john9527 the dns server should still resolve the ip that should still work as intended on your fork right ?
Click to expand...
Yes, unless the hostip that's included with dnscrypt has a different syntax from the entware version (probably not) or there is a mismatch somehow in any linked libraries.
 
You must log in or register to reply here.

Similar threads

Xentrk
ipset example script on GitHub..Edits made for arm/mips syntax
Replies
0
Views
2K
Xentrk
Xentrk
Xentrk
Selective Routing over OpenVPN tunnel gone awry
Replies
11
Views
3K
Xentrk
Xentrk
Similar threads
Thread starter Title Forum Replies Date
M RT-AX86U and DNS Privacy over USB WAN Asuswrt-Merlin 0
T Merlin closed source parts and privacy questions Asuswrt-Merlin 15
TheLyppardMan MAC Filter Options Asuswrt-Merlin 29
M Wireless Mac filter defaulting to whitelist. Asuswrt-Merlin 5
R Wireless MAC (Band filter filtering out Guest network) Asuswrt-Merlin 2
G Feature request: firewall url filter log Asuswrt-Merlin 0
TanyaC cannot add domain to router URL filter Asuswrt-Merlin 15
C How to filter out foreign countries Asuswrt-Merlin 3
P Using Avahi to cast to a PyChromeCast on another subnet Asuswrt-Merlin 4
S Transfer files to another device (raspi) in the same network Asuswrt-Merlin 3

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 701 (members: 15, guests: 686)
Top