Malware Filter / bad host IPSET

[swetoast]

still dont like that site. i used the wiki (only!) dont rely on guides the wiki is maintained by @RMerlin and other volunteers like me if any major changes occur the wiki is always updated and if you really want to get into it then you got the entware guys @zyxmon and @ryzhov_al. They also have a wiki at their gitlab. So using a guide like that ro site is last case scenario since its outdated, when you got better methods right in front of you.

 

[visortgw]

What happened to rev 15? I received an e-mail notification that it was posted, but it does not appear anywhere that I can find...

 

[swetoast]

working on it made more modifications so it will be out soon, tomsk pointed out some stuff in another thread so im fixing some issues

 

[swetoast]

So here it is revision 15

• Cleanup procedure
• Path change (no more /opt)

Note: you can use your old path if you want i only changed path so that new inexperienced users dont have to set a path so if your stuff is still in opt then use opt else it will failover and download the list automatically

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 15

path=/tmp/malware-filter                                # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;
*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
if [ ! -f $path/malware-filter.list ]
then wget $url -O $path/malware-filter.list; get_source; else get_source; fi }

check_path () {
if [ ! -d "$path" ]; then
     mkdir /tmp/malware-filter
     path='/tmp/malware-filter'
     echo "path is not found so we created $path now getting the list..."
     check_failover
else check_failover; fi }

check_failover () {
if [ ! -d "$path" ]; then
     echo "failed to set failover path"
     exit 1
else get_list; fi }

get_source () {
    mkdir -p $path
    wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.tmp
    awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' $path/malware-list.tmp > $path/malware-list.pre
    cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-list.blocklist
}

run_ipset () {

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-list.blocklist nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat $path/malware-list.blocklist | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-list.blocklist nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat $path/malware-list.blocklist | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi }

cleanup () {
logger -s -t system "Malware Filter loaded $(cat $path/malware-list.blocklist | wc -l) unique ip addresses."
find $path -type f -name 'malware-list.*' -delete
}

check_path
run_ipset
cleanup

exit $?

 

[swetoast]

Revision 16
Changelog:

• no more opt
• using tmp for downloads
• online check (doesn't run if no net is detected)

note: this revision will make it easier for newbies to use this script all they have to do is place malware-block somewhere on the jffs partition and run it and it will sort itself out.

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 16

blocklist=/jffs/malware-filter.list                     # Set your path here
interface=$(nvram get wan0_ifname)                      # Change this value if this is not your internet interface
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers
    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'
     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;
*v4) # Value for Mips Routers
    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''
    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

check_online () {
iface=`grep "$interface" /proc/net/dev`
if  [ -n "$iface" ]; then
    if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp") ]
    then get_list; fi
    else exit 1; fi
}

get_list () {
url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
if [ ! -f $blocklist ]
then wget $url -O $blocklist; get_source; else get_source; fi }

get_source () {
    wget -q --tries=$retries --show-progress -i $blocklist -O $TMP/malware_filter_raw.part
    awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' $TMP/malware_filter_raw.part > $TMP/malware_filter_presort.part
    cat $TMP/malware_filter_presort.part | grep -oE "$regexp" | sort -u > $TMP/malware_filter_sorted.part
}

run_ipset () {
echo "adding ipset rule to firewall this will take time."
ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $TMP/malware_filter_sorted.part nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat $TMP/malware_filter_sorted.part | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $TMP/malware_filter_sorted.part nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat $TMP/malware_filter_sorted.part | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi
iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi }

cleanup () {
logger -s -t system "Malware Filter loaded $(cat $TMP/malware_filter_sorted.part | wc -l) unique ip addresses."
find $TMP -type f -name 'malware_filter_*.part' -delete
}

check_online
run_ipset
cleanup

exit $?

 

[Martineau]

interface=eth0                    # Change this value if this is not your internet interface

Is this not easier for newbies?

interface=$(nvram get wan0_ifname)   # Change this value if this is not your internet interface

 

[swetoast]

true, havent messed around with nvram that much so i dont know how much it can handle, so tnx ill make the changes

 

[swetoast]

the only issue with that line is that now that if i run this on a non asus distro there must be a failover interface in place not intending for my scripts to be limited to just asuswrt

check_online () {
if [ -z "$(which nvram)" ]; then
iface=`grep "$failover" /proc/net/dev`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
else iface=`nvram get wan0_ifname`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "$regexp_v4") ]
     then get_list; fi
     else exit 1; fi
fi }

 

[tomsk]

I used

 $(nvram get wan0_gw_ifname)

in one of my scripts because i noticed the interface would flip between vlan2 and eth0 depending if Aiprotection was switched on.
which nvram value is better for this case?

 

[swetoast]

@RMerlin which one is better ? i truly at a lost here cause i haven't messed around with nvram

 

[shooter40sw]

Hi! does it have to be on the jffs? cant I just place it anywhere on the usb drive y point it there in the script?
Thanks

So here it is revision 15

• Cleanup procedure
• Path change (no more /opt)

Note: you can use your old path if you want i only changed path so that new inexperienced users dont have to set a path so if your stuff is still in opt then use opt else it will failover and download the list automatically

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 15

path=/tmp/malware-filter                                # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;
*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
if [ ! -f $path/malware-filter.list ]
then wget $url -O $path/malware-filter.list; get_source; else get_source; fi }

check_path () {
if [ ! -d "$path" ]; then
     mkdir /tmp/malware-filter
     path='/tmp/malware-filter'
     echo "path is not found so we created $path now getting the list..."
     check_failover
else check_failover; fi }

check_failover () {
if [ ! -d "$path" ]; then
     echo "failed to set failover path"
     exit 1
else get_list; fi }

get_source () {
    mkdir -p $path
    wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.tmp
    awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' $path/malware-list.tmp > $path/malware-list.pre
    cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-list.blocklist
}

run_ipset () {

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-list.blocklist nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat $path/malware-list.blocklist | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-list.blocklist nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat $path/malware-list.blocklist | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi }

cleanup () {
logger -s -t system "Malware Filter loaded $(cat $path/malware-list.blocklist | wc -l) unique ip addresses."
find $path -type f -name 'malware-list.*' -delete
}

check_path
run_ipset
cleanup

exit $?

 

[swetoast]

yes, just revision 16 not 15, only using jffs since it simplifies things for me. Basically you can use /opt/bin/malware-block like before with the list there was as well all you have to do is change path on blocklist

 

[Martineau]

the only issue with that line is that now that if i run this on a non asus distro there must be a failover interface in place not intending for my scripts to be limited to just asuswrt

Ahh, silly me for posting use of nvram in the Asus-Merlin forum.

Not sure if this is reliable (e.g. Asus-Merlin with Dual Wan may be incorrect?)

ip neighbour show | grep -vE "^192|10|172" | cut -d" " -f3

 

[swetoast]

Well it might be posted in Asuswrt-Merlin subforum but i want it to have compatibility with other brands also (dont think that a huge deal) its in there just an if sentence around it that if nvram is not present

 

[RMerlin]

@RMerlin which one is better ? i truly at a lost here cause i haven't messed around with nvram

There's no single answer to this, as it depends on the type of WAN, also whether you use dual wan or not.

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/shared/rtstate.c#L155

So it can be wan0_ifname, or wan0_pppoe_ifname. And if you use dual WAN, then unit 1 would be wan1_*.

 

[Xentrk]

Thank you @swetoast for the kewl scripts you have developed and maintain. I will send you a donation when I return from holiday. I support a D-Link flashed with dd-wrt at a children's home. A dd-wrt forum member supplied me with the ipset module that I can compile on the D-Link. When I get back home, I plan to compile it and see if I can get this and the privacy-filter scripts running on the D-Link as well. I will let you know how it turns out.

 

[swetoast]

should work there is nothing stopping it, if there is let me know cause my plan is too have it cross platform.

 

[shooter40sw]

Hi @swetoast Im having an issue with version 16, when I manually run the script it works as intended, but when the cron job runs, it does not populate the ipset, the only change I made was the path, that I dont have the filter in the jffs but on the same route that I made.

If I manually run it populates it but the cron job deletes the set after it runs.

/tmp/mnt/sda1/malware-filter# tail -f /tmp/syslog.log
Feb 20 01:00:00 crond[247]: USER  pid 7886 cmd /tmp/mnt/sda1/malware-filter/malware-block
Feb 20 01:00:05 system: Malware Filter loaded 0 unique ip addresses.

the code to call the cron is

cru a malware-update "00 01 * * * /tmp/mnt/sda1/malware-filter/malware-block"

I run manually the script 2 time and in works normally just testing if running it twice was going to delete the set, so I really dont understand

Thanks

 

[swetoast]

Will take a peak at it after work

 

[HardCat]

Version 16 working well here, this is the cron being added in /jffs/scripts/services-start

cru a malware-filter "0 */12 * * * /opt/bin/malware-block"

Cheers...