Menu
What's new
By:
Filters Better Search

Using ipset revisited

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

N

Neurophile

Occasional Visitor
As part of a round of beefing up the security of my network I decided to set up ipset/iptables to filter traffic from the darkest corners of the interwebs. I started with the Malware Filter script from the wiki here: https://github.com/RMerl/asuswrt-merlin/wiki/Using-ipset
Unfortunately, using 380.58 on an N66U this script is broken...so I fixed it and tweaked it to my needs. Here is the result.

First create the scripts (full code can be found on my github: https://github.com/Neurophile/ASUSWRT-scripts )
1) /jffs/scripts/firewall-start
2) /jffs/scripts/firewall-update
3) /jffs/scripts/services-start

set each one executable with, e.g.: chmod +x /jffs/scripts/firewall-start

Next create the list of blacklists that you want to use:
/jffs/filters/malware-filter.list

Mine is a very minimal set for now, more are listed in my master-blacklist-list or use your own favorites. Manually run the firewall-update script once to create and populate the IP blocklist cache file /jffs/filters/malware-filter.txt

You can now run firewall-start or reboot the router and your new blocklist will be blocking! The blacklist updates every night at 3 am via the cron job setup in the services-start script.

Enjoy!
 
Neurophile said:
Next create the list of blacklists that you want to use:
/jffs/filters/malware-filter.list

Mine is a very minimal set for now, more are listed in my master-blacklist-list or use your own favorites. Manually run the firewall-update script once to create and populate the IP blocklist cache file /jffs/filters/malware-filter.txt

You can now run firewall-start or reboot the router and your new blocklist will be blocking! The blacklist updates every night at 3 am via the cron job setup in the services-start script.

Enjoy!
Click to expand...


Haha nice of ya to use my script tnx :D hope that it works out for ya and tnx for the revision
feel free to update the wiki both me and rmerlin would be thankful :)
 
swetoast said:
Haha nice of ya to use my script tnx :D hope that it works out for ya and tnx for the revision
feel free to update the wiki both me and rmerlin would be thankful :)
Click to expand...

Thanks for writing the original! The main failure is that wget doesn't recognize the ssl certificates. I hacked around this by adding --no-check-certificate but that is not the safest fix. I am 99.9% sure that spamhaus has a valid ssl, so it has to be something with either the wget config or with ssl certs as a whole in the router. If I can figure that out I will fix up the wiki.
 
Neurophile said:
Thanks for writing the original! The main failure is that wget doesn't recognize the ssl certificates. I hacked around this by adding --no-check-certificate but that is not the safest fix. I am 99.9% sure that spamhaus has a valid ssl, so it has to be something with either the wget config or with ssl certs as a whole in the router. If I can figure that out I will fix up the wiki.
Click to expand...

Wget does not let you specify the location of the CA bundle at compile time, it's hardcoded to the default location used by OpenSSL. Unfortunately, Asus decided to use that default location to store the httpd's SSL certificate, making it hard to include a default bundle without having to shuffle a lot of things around.

As a workaround, you can use curl - curl is compiled to point at the CA bundle stored in /rom/ .
 
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.
 
Neurophile said:
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.
Click to expand...

I'll take a look to see if it could easily be done in a global wgetrc file.
 
This setup works with my RT-AC87U. Using both malware-filter and windows 10 filter. Crazy number of ips being blocked.
 
FYI if you want to see how much (if any) traffic is being blocked by the list do:
iptables -L -v | less

up near the top of the first page you should see something like this
Code: 
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source        destination         
 4    336  DROP       all  --  any    any     anywhere      anywhere       set malware-filter dst
This is what it looked like after I sent 4 ping packets to an ip on the list to make sure it is working.
 
wget will look for certs in the directory pointed at by the SSL_CERT_DIR env variable. I use the little snippet below to make wget work with ssl in my scripts

Code: 
# Define path to certs so HTTPS works in wget
# Certificates are installed using 'opkg install ca-certificates'
export SSL_CERT_DIR=/opt/etc/ssl/certs
 
Nice too see my little script benefit so many :)

yepp and i can agree to how many ips are being banned but the net is a shirtty place with spammer and malware spreaders so why not try and make the most of it and ban em :)
 
Neurophile said:
OK. I think I have it fixed now. Thanks to this thread: http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/
I added a line to services-start to update the user .wgetrc at boot time to point to the cert bundle.
echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

A more permanent fix would be to add a system level wget config file at /usr/local/etc/wgetrc to the distro, but I have no idea if that makes sense given the tiny fraction of the user base that would benefit.

I can switch to curl if there is a compelling reason to, but this seems to be working now.
Click to expand...

Fixed.
 
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.

One is for the testing of which version of ipset is running for the loading of the modules that @john9527 has pointed out in his thread http://www.snbforums.com/threads/fo...ilable-v18e3-v19b4.18914/page-209#post-278253

The other is where the script tests for an "Uknown set"

Code: 
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep 'Unknown set')" != "" ]; then

We can change it to test for both version 4 and version 6 with this small change.

Code: 
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then

Cheers.
 
HardCat said:
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.

One is for the testing of which version of ipset is running for the loading of the modules that @john9527 has pointed out in his thread http://www.snbforums.com/threads/fo...ilable-v18e3-v19b4.18914/page-209#post-278253

The other is where the script tests for an "Uknown set"

Code: 
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep 'Unknown set')" != "" ]; then

We can change it to test for both version 4 and version 6 with this small change.

Code: 
if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then

Cheers.
Click to expand...

Thanks very much. This info was very helpful.

Huge thanks and shutout to @HardCat

All working now as expected, I need to read the change log more often, I guess I get complacent at times, which I can only blame myself :rolleyes:
 
HardCat said:
Merlin has recently (380.63) updated ipset to version 6 for the arm routers, this has required some changes to the firewall start script for two reasons that I have found so far.
Click to expand...
Regarding the new ipset 6.x version, I'm still figuring ipset out, and I'm using the sample scripts on the RMerlin wiki here, but I'm not sure yet what I really need to all change to get it working. Could someone point me in the correct direction, or update the wiki?

At the moment, this is all i get from the first script sample:
Code: 
./firewall-start
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: 'ip_set_nethash.ko': module not found
insmod: 'ip_set_iphash.ko': module not found
insmod: 'ipt_set.ko': module not found
Bad argument `TorNodes'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `BlockedCountries'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `MicrosoftSpyServers'
Try `iptables -h' or 'iptables --help' for more information.

Current script sample:
Code: 
#!/bin/sh

# Loading ipset modules
lsmod | grep "ipt_set" > /dev/null 2>&1 || \
for module in ip_set ip_set_nethash ip_set_iphash ipt_set
do
   insmod $module
done

# Preparing folder to cache downloaded files
IPSET_LISTS_DIR=/jffs/ipset_lists
[ -d "$IPSET_LISTS_DIR" ] || mkdir -p $IPSET_LISTS_DIR

# Different routers got different iptables syntax
case $(uname -m) in
  armv7l)
   MATCH_SET='--match-set'
   ;;
  mips)
   MATCH_SET='--set'
   ;;
esac

# Block traffic from Tor nodes
if [ "$(ipset --swap TorNodes TorNodes 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N TorNodes iphash
   [ -e $IPSET_LISTS_DIR/tor.lst ] || wget -q -O $IPSET_LISTS_DIR/tor.lst http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
   for IP in $(cat $IPSET_LISTS_DIR/tor.lst)
   do
       ipset -A TorNodes $IP
   done
fi
[ -z "$(iptables-save | grep TorNodes)" ] && iptables -I INPUT -m set $MATCH_SET TorNodes src -j DROP

# Block incoming traffic from some countries. cn and pk is for China and Pakistan. See other countries code at http://www.ipdeny.com/ipblocks/
if [ "$(ipset --swap BlockedCountries BlockedCountries 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N BlockedCountries nethash
   for country in pk cn
   do
       [ -e $IPSET_LISTS_DIR/$country.lst ] || wget -q -O $IPSET_LISTS_DIR/$country.lst http://www.ipdeny.com/ipblocks/data/countries/$country.zone
       for IP in $(cat $IPSET_LISTS_DIR/$country.lst)
       do
           ipset -A BlockedCountries $IP
       done
   done
fi
[ -z "$(iptables-save | grep BlockedCountries)" ] && iptables -I INPUT -m set $MATCH_SET BlockedCountries src -j DROP

# Block Microsoft telemetry spying servers
if [ "$(ipset --swap MicrosoftSpyServers MicrosoftSpyServers 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
   ipset -N MicrosoftSpyServers iphash
   for IP in 23.99.10.11 63.85.36.35 63.85.36.50 64.4.6.100 64.4.54.22 64.4.54.32 64.4.54.254 \
             65.52.100.7 65.52.100.9 65.52.100.11 65.52.100.91 65.52.100.92 65.52.100.93 65.52.100.94 \
             65.55.29.238 65.55.39.10 65.55.44.108 65.55.163.222 65.55.252.43 65.55.252.63 65.55.252.71 \
             65.55.252.92 65.55.252.93 66.119.144.157 93.184.215.200 104.76.146.123 111.221.29.177 \
             131.107.113.238 131.253.40.37 134.170.52.151 134.170.58.190 134.170.115.60 134.170.115.62 \
             134.170.188.248 157.55.129.21 157.55.133.204 157.56.91.77 168.62.187.13 191.234.72.183 \
             191.234.72.186 191.234.72.188 191.234.72.190 204.79.197.200 207.46.223.94 207.68.166.254
   do
       ipset -A MicrosoftSpyServers $IP
   done
fi
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
Last edited:
octopus said:
http://www.snbforums.com/threads/using-ipset-revisited.32817/#post-287431
http://www.snbforums.com/threads/fo...lts-releases-v20e9.18914/page-212#post-278873
Click to expand...
Ah, thanks! The second link was the one I couldn't find! :)

Still having a problem though...
Code: 
Bad argument `TorNodes'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `BlockedCountries'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `MicrosoftSpyServers'
Try `iptables -h' or 'iptables --help' for more information.
But if I run the commands manually, they all seem to work. Anyone see anything wrong?

Code: 
#!/bin/sh

# Loading ipset modules
ipset -v | grep -i "v4" > /dev/null 2>&1                                      
if [ $? -eq 0 ];                                                              
then                                                                    
   # old ipset                                                          
   ipsetv=4                                                              
   lsmod | grep "ipt_set" > /dev/null 2>&1 || \                          
   for module in ip_set ip_set_nethash ip_set_iphash ipt_set            
   do                                                                    
       insmod $module                                                
   done                                                                  
else                                                                          
   # new ipset                                                  
   ipsetv=6                                                    
   lsmod | grep "xt_set" > /dev/null 2>&1 || \                  
   for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set  
   do                                                          
       insmod $module                                      
   done                                                        
fi        

# Preparing folder to cache downloaded files
IPSET_LISTS_DIR=/jffs/ipset_lists
[ -d "$IPSET_LISTS_DIR" ] || mkdir -p $IPSET_LISTS_DIR

# Different routers got different iptables syntax
case $(uname -m) in
  armv7l)
    MATCH_SET='--match-set'
    ;;
  mips)
    MATCH_SET='--set'
    ;;
esac

# Block traffic from Tor nodes
if [ "$(ipset --swap TorNodes TorNodes 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create TorNodes hash:ip
    [ -e $IPSET_LISTS_DIR/tor.lst ] || wget -q -O $IPSET_LISTS_DIR/tor.lst http://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv
    for IP in $(cat $IPSET_LISTS_DIR/tor.lst)
    do
        ipset add TorNodes $IP
    done
fi
[ -z "$(iptables-save | grep TorNodes)" ] && iptables -I INPUT -m set $MATCH_SET TorNodes src -j DROP

# Block incoming traffic from some countries. cn and pk is for China and Pakistan. See other countries code at http://www.ipdeny.com/ipblocks/
if [ "$(ipset --swap BlockedCountries BlockedCountries 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create BlockedCountries hash:net
    for country in pk cn
    do
        [ -e $IPSET_LISTS_DIR/$country.lst ] || wget -q -O $IPSET_LISTS_DIR/$country.lst http://www.ipdeny.com/ipblocks/data/countries/$country.zone
        for IP in $(cat $IPSET_LISTS_DIR/$country.lst)
        do
            ipset add BlockedCountries $IP
        done
    done
fi
[ -z "$(iptables-save | grep BlockedCountries)" ] && iptables -I INPUT -m set $MATCH_SET BlockedCountries src -j DROP

# Block Microsoft telemetry spying servers
if [ "$(ipset --swap MicrosoftSpyServers MicrosoftSpyServers 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]
then
    ipset create MicrosoftSpyServers hash:ip
    for IP in 23.99.10.11 63.85.36.35 63.85.36.50 64.4.6.100 64.4.54.22 64.4.54.32 64.4.54.254 \
              65.52.100.7 65.52.100.9 65.52.100.11 65.52.100.91 65.52.100.92 65.52.100.93 65.52.100.94 \
              65.55.29.238 65.55.39.10 65.55.44.108 65.55.163.222 65.55.252.43 65.55.252.63 65.55.252.71 \
              65.55.252.92 65.55.252.93 66.119.144.157 93.184.215.200 104.76.146.123 111.221.29.177 \
              131.107.113.238 131.253.40.37 134.170.52.151 134.170.58.190 134.170.115.60 134.170.115.62 \
              134.170.188.248 157.55.129.21 157.55.133.204 157.56.91.77 168.62.187.13 191.234.72.183 \
              191.234.72.186 191.234.72.188 191.234.72.190 204.79.197.200 207.46.223.94 207.68.166.254
    do
        ipset add MicrosoftSpyServers $IP
    done
fi
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
I see you have already modified script to match ipset 6.29.
Try to remove "--" before swap...........

Can you see your rule in INPUT, FORWARD chain?
Code: 
iptables -vnL INPUT
iptables -vnL FORWARD

Remove possible "CR"
Code: 
dos2unix /jffs/scripts/firewall-start
 
octopus said:
I see you have already modified script to match ipset 6.29.
Try to remove "--" before swap...........

Can you see your rule in INPUT, FORWARD chain?
Code: 
iptables -vnL INPUT
iptables -vnL FORWARD

Remove possible "CR"
Code: 
dos2unix /jffs/scripts/firewall-start
Click to expand...

Nope same problem, and no the rules don't show up. However, after I've manually run the below once for each of the three, the script works fine.
Code: 
MATCH_SET='--match-set'
[ -z "$(iptables-save | grep MicrosoftSpyServers)" ] && iptables -I FORWARD -m set $MATCH_SET MicrosoftSpyServers dst -j DROP
 
Last edited:
You must log in or register to reply here.

Similar threads

T
Troubles with VPN Client + DDNS Setup on GT-BE98
Replies
5
Views
358
Tuntenfisch
T
R
DNS filtering when using cloudflare DNS?
Replies
6
Views
1K
Crimliar
Crimliar
r000t
Hardcoded Google DNS IPTABLES rule
Replies
6
Views
423
r000t
r000t
M
deSEC DDNS update script
Replies
4
Views
950
klcb2010
K
N
RT-AX57 GO - can't add my script/service to autorun
Replies
9
Views
871
Grumpy_Geoff
G
Similar threads
Thread starter Title Forum Replies Date
P Accessing remotly Server While Using VPN on Asus Router with Merlin Firmware Asuswrt-Merlin 9
Zakalwe Solved RT-AX86U Pro - Using as PPPoE modem with BT Full Fibter internet Asuswrt-Merlin 8
G Using VPN Director to route only torrent traffic through VPN Asuswrt-Merlin 9
F Unbricking RT-AC5300 using Serial Asuswrt-Merlin 3
T Wireguard Client VPN not using DNS Asuswrt-Merlin 14
R Connecting to the router using ddns domain name from within the network? Asuswrt-Merlin 15
S Is it possible to use 5Ghz-2 on the XT8 when using it as an AiMesh node with a dual band main router? Asuswrt-Merlin 1
J SSID QR Code validity after hard reset but using same SSID Password when setup again? Asuswrt-Merlin 1
N HTTPS/SSL Certificate issue when using WWW.NAMECHEAP.COM in DDNS Asuswrt-Merlin 4
Y How to restart dropbear using cru Asuswrt-Merlin 2

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 629 (members: 29, guests: 600)
Top