Malware Filter / bad host IPSET

[wallyg8r]

just outta curiosity how much is your malware filter catching ?

ipset -L malware-filter | wc -l

i got 7 lists in my filter and my block is ~49K of blocked IP to be specific 49815

just remember that its minus 7 of the given value since its seven lines of text that gets counted into the wc -l command

Mine is 34295. Thanks for the script.

 

[swetoast]

recently broke a new record

Mar 2 12:07:14 system: Malware Filter loaded 52282 unique ip addresses.

freaking insane numbers of infected or infecting clients outthere

 

[Cedarhillguy]

I tried upgrading to version 17, but doesn't work as-is for a 68u, version 380.65 without entware. It reports the find command, used near the end for cleanup, doesn't support -type and -delete arguments. I was able to get the find command to work using the below code.

find /tmp -name 'malware-filter-*.part' -exec rm {} +

I would also suggest for consideration changing the logger to show the actual number of addresses in IPSET. Current statement assumes everything in the file is loaded, which may not always be the case. The below, I believe would show what is actually in the IPSET tables.

logger -s -t system "Malware Filter loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."

 

[swetoast]

kewl, ill do some testing and if it work for me aswell then ill merge it

 

[swetoast]

Boom revision bump due to incompability without entware, tnx Cedarhillguy
it also it states the correct number for ips blocked in syslog.

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife, visortgw, Cedarhillguy
# Testers: shooter40sw
# Revision 18
blocklist=/jffs/malware-filter.list                     # Set your path here
failover=eth0                                           # Change only if WAN interface is not detected.
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value
case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers
    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'
     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;
*v4) # Value for Mips Routers
    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''
    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac
check_online () {
if [ -z "$(which nvram)" ]; then
iface=`grep "$failover" /proc/net/dev`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
     then get_list; fi
     else exit 1; fi
else iface=`nvram get wan0_ifname`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
     then get_list; fi
     else exit 1; fi
fi }
get_list () {
url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
if [ ! -f $blocklist ]
then wget $url -O $blocklist; get_source; else get_source; fi
}
get_source () {
    wget -q --tries=$retries --show-progress -i $blocklist -O /tmp/malware-filter-raw.part
        awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' /tmp/malware-filter-raw.part > /tmp/malware-filter-presort.part
    cat /tmp/malware-filter-presort.part | grep -oE "$regexp" | sort -u > /tmp/malware-filter-sorted.part
}
run_ipset () {
echo "adding ipset rule to firewall this will take time."
ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi
iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi }
cleanup () {
logger -s -t system "Malware Filter loaded $(ipset -L malware-filter | wc -l | awk '{print $1-7}') unique ip addresses."
find /tmp -name 'malware-filter-*.part' -exec rm {} +
}
check_online
run_ipset
cleanup
exit $?

 

[visortgw]

I have been experiencing inconsistent results with the check_online routine, where it does not always report the IP address to confirm online status. I have changed to the following (Thank you, Google!):

if [ "$(ip addr show eth0 | grep inet | awk '{print $2}' | sed 's#/.*##')" \
!= "" ] then get_list;
else exit 1; fi​

 

[redhat27]

I've used [ $(nvram get wan0_state_t) -eq 2 ] to check if router is online in the past. Not sure if that is correct...

 

[visortgw]

I've used [ $(nvram get wan0_state_t) -eq 2 ] to check if router is online in the past. Not sure if that is correct...

That appears to be what has been used in rstats source. It also seems to be quickest method to verify WAN connection status.

 

[Xentrk]

ipset on DD-WRT

Hi @swetoast,

I know you are trying to make the malware-filter and privacy-filer generic. So I wanted to see if I could get them to work on DD-WRT.

I got ipset to work on DD-WRT build Firmware: DD-WRT v3.0-r29837 std (06/06/16). ipset is not included in dd-wrt builds and the iptables version is old. To get it working, I followed the instructions here http://www.dd-wrt.com/phpBB2/viewtopic.php?p=1035801 with the caveat of compiling the included xp_set.ko file.

mkdir /jffs/usr
cd /jffs/usr
wget -O ipset_ipt_libmnl.K3.Arm.tar http://www.dd-wrt.com/phpBB2/download.php?id=31626
wget -O dnsmasq_ipset.tar http://www.dd-wrt.com/phpBB2/download.php?id=35637
tar xf ipset_ipt_libmnl.K3.Arm.tar
tar xf dnsmasq_ipset.tar
rm ipset_ipt_libmnl.K3.Arm.tar
rm dnsmasq_ipset.tar

Then, get the xp_set.zip file from here:
https://www.dd-wrt.com/phpBB2/viewtopic.php?t=306131

Download to PC and unzip. Copy xt_set.ko to /jffs/usr/lib/modules/

insmod /jffs/usr/lib/modules/xt_set.ko

# ipset -v
ipset v6.21.1, protocol version: 6

#lsmod | grep xt_set
xt_set                  6435  0 [permanent]

I tried to use the version if ipset on entware. But it is very old and did not work. So I removed the package. Following is the version information. Entware could use an update on ipset!

ipset v4.5, protocol version 4.
ipset v4.5: Kernel ip_set module is of protocol version 6.I'm of protocol version 4.
Please upgrade your kernel and/or ipset(8) utillity.

The iptables that came with the above tar files and installed in jffs/usr/sbin is dated as well

# iptables -v
iptables v1.3.7: no command specified

So, I installed iptables from entware.

#opkg install iptables
[CODE]# /opt/sbin/iptables -v
iptables v1.4.21: no command specified

But I had to change my PATH to make sure the script used the newer version:

PATH=/opt/sbin:/opt/bin:/bin:/bin:/usr/bin:/sbin:/usr/sbin:/jffs/usr/sbin:/jffs/usr/mmc/sbin:/mmc/bin:/mmc/usr/sbin:/mmc/usr/bin

This allowed me to run the Tor and Countries Block script. I then added the following to the Administration - Commands – Startup so it would start the script at boot up:

insmod /jffs/usr/lib/modules/xt_set.ko
PATH=/opt/sbin:/opt/bin:/bin:/bin:/usr/bin:/sbin:/usr/sbin:/jffs/usr/sbin:/jffs/usr/mmc/sbin:/mmc/bin:/mmc/usr/sbin:/mmc/usr/bin
sh /jffs/scripts/create-ipset-lists.sh

~# iptables -L | grep set
DROP       all  --  anywhere             anywhere             match-set BlockedCountries src
DROP       all  --  anywhere             anywhere             match-set TorNodes src
DROP       all  --  anywhere             anywhere             match-set MicrosoftSpyServers dst
REJECT     tcp  --  anywhere             anywhere             reject-with tcp-reset

I then tried to run malware-block and privacy-filter. But they both exit on the check_online function. I wonder if you can help me figure out why. Is it because my NVRAM is limited? I have 64KB total of NVRAM and 46KB is being used. I don’t have a /usr/sbin/nvram folder so 0 length gets returned on that line.

check_online
+ check_online
+ which nvram
+ [ -z /usr/sbin/nvram ]
+ nvram get wan0_ifname
+ iface=
+ [ -n  ]
+ exit 1

/jffs/scripts # grep "eth0" /proc/net/dev
  eth0: 1299364610 1292030    0  48    0     0          0         0 1221297441 1254742    0    0    0     0       0          0[\CODE]

/jffs/scripts # nvram show | grep wan0_ifname
size: 46387 bytes (19149 left)

 

[redhat27]

But they both exit on the check_online function.

Maybe you can try swapping the check_online function with:

check_online () {
  [ $(nvram get wan0_state_t) -eq 2 ] && get_list || exit 1
}

 

[Xentrk]

Maybe you can try swapping the check_online function with:

check_online () {
  [ $(nvram get wan0_state_t) -eq 2 ] && get_list || exit 1
}

Thanks for the suggestion. I get an unkown operand error with the revised function. I will continue to analyze..

check_online
+ check_online
+ nvram get wan0_state_t
+ [ -eq 2 ]
sh: 2: unknown operand
+ exit 1

At the suggestion of https://www.shellcheck.net/#, I quoted the nvram:

check_online () {
  [ "$(nvram get wan0_state_t)" -eq 2 ] && get_list || exit 1
}

I now get

+ check_online
+ nvram get wan0_state_t
+ [  -eq 2 ]
sh: out of range
+ exit 1

Getting closer, on AC88U, I get two lines returned:

nvram show | grep wan0_state_t
wan0_state_t=2
size: 60870 bytes (70202 left)

On dd-wrt, only the size information is returned. I need to find the equivalent nvram parameter in dd-wrt. Getting closer..

I found the equivalent nvram parameter in dd-wrt. It is "wanup". It was set to "1" so that is what I used in the check:

}
check_online () {
  [ "$(nvram get wanup)" -eq 1 ] && get_list || exit 1
}

I made the same change to privacy-filter and that worked too. So this is very kewl!!! I have been working on this for 5 days off and on and it is good to see it working.

# iptables -L | grep match
DROP       all  --  anywhere             anywhere             match-set BlockedCountries src
DROP       all  --  anywhere             anywhere             match-set TorNodes src
REJECT     all  --  anywhere             anywhere             match-set privacy-filter_ipv4 src,dst reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             match-set malware-filter src,dst reject-with icmp-port-unreachable
DROP       all  --  anywhere             anywhere             match-set MicrosoftSpyServers dst

DD-WRT does not make this simple. Not including ipset and having old iptables version in the kernel does not help. Flashing any newer firmware past June 2016 bricks the D-Link 880L. Many guys complaining about this in the dd-wrt Broadcom Forum, D-Link 880L thread. I had to use my USB to TTL cable more than once to unbrick when trying newer dd-wrt versions. Entware came through on providing updated iptables. But the ipset version on entware could use an update.

I may post my findings on the dd-wrt.com forum pending feedback from @swetoast. This was done using privacy-filter version 12 and version 18 on malware-filter.

The children's home where this router is installed is going to let me replace this router with another. It will definitely be an ASUS. I will then turn the D-Link into an AP to give improve wifi range at the home. I just could not resist the challenge to make these scripts work on another platform.

 

[redhat27]

Yes, sorry. It appears your dd-wrt router does not have that variable defined in nvram, here is an alternate way:

check_online () {
  ping -q -c 1 google.com >/dev/null 2>&1 && get_list || exit 1
}

 

[redhat27]

@swetoast Do you know why the checks for nvram in path or checking if router is online are even necessary? Shouldn't all asuswrt routers have that and be expected to always be online? I mean if the router is not online, that's a major issue.

 

[redhat27]

I found the equivalent nvram parameter in dd-wrt. It is "wanup".

Have you tested this? Quick test would be to disconnect the wan ethernet and checking if that flips.

 

[Xentrk]

Have you tested this? Quick test would be to disconnect the wan ethernet and checking if that flips.

Excellent point @redhat27! I will test it tomorrow and let you know.

 

[swetoast]

Hi guys, been at work and i see you got some suggestions and im sure those are excellent so im gonna make the changes when i got some time over based on your suggestions cause i want to work with *wrt and x86 out of the box

 

[Xentrk]

I can help test on wrt platform for you. From what I read on the forums, ipset is not included in the dd-wrt builds to save space. And, the iptables verison on dd-wrt is also old. Having ipset version updated on entware will allow someone to easily get around the issues I had. I wonder if @ryzhovau can update ipset4 entware package to more recent version? And maybe iptables as well?

 

[skeal]

This is great it installed on my AC68U real well. Great script!

 

[swetoast]

tnx @skeal, feel free to check out my other scripts

 

[Xentrk]

Excellent point @redhat27! I will test it tomorrow and let you know.

I confirmed the DD-WRT nvram parameter wanup=1 when the WAN is UP and wanup=0 when the WAN is DOWN.