Malware Filter / bad host IPSET

[HardCat]

Results as follows: RT-AC87R Entware installed.

system: Malware Filter loaded 23919 unique ip addresses.
real 2m 13.30s
user 2m 0.46s
sys 1m 56.97s

 

[shooter40sw]

So ive been running test since the new revision and 30000+ takes around 5~ min on a RT-AC56U kinda curious to hear results from others. I am interested in both routers with and without entware installed.

just do this command

time malware-block

and when the script ends it prints out 3 stats i used the "real" as measurement for my tests so please post avg time and router model and if entware was present.

Hi, I have not installed this version 10, I have 9 installed, that commando does not work for me, but the script is working and the ipset full.

/cache/malware-filter# time malware-block
time: can't execute 'malware-block': No such file or directory
Command exited with non-zero status 127
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s

And that is the name of the script, yesterday I factory reset the router and configured everything from scratch
regards

 

[Xentrk]

The "time" command did not work. I then installed the entware package time:

opkg install time

My results are as follows:

system: Malware Filter loaded 35002 unique ip addresses.
159.67user 107.07system 2:48.78elapsed 158%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps

I reran as follows to make the output easier to read:

time -f "\t%U user,\t%S system,\t%E elasped, \t%e real" malware-block

And the output is:

system: Malware Filter loaded 34997 unique ip addresses.
        161.19 user,    105.15 system,  2:45.65 elasped,        165.65 real

 

[Cedarhillguy]

I have an RT-AC68U (800mhz CPU), no entware installed. Script version 9.
Running the time /jffs/scripts/malware-block manually reports:
Real: 6m 24.48s Loaded 34,977 IP addresses.

I've been running a slightly modified script that captures the begin/end times and records the total execution seconds in the system log. Here are some states from previous log entries.
40245 loaded 7min 33sec (453sec)
38990 loaded 7min 16sec (436sec)
32267 loaded 6min 15 sec (375sec)
32094 loaded 5min 52 sec (352sec)
31188 loaded 5min 45 sec (345sec)
35016 loaded 6min 21 sec (381sec)

 

[visortgw]

So ive been running test since the new revision and 30000+ takes around 5~ min on a RT-AC56U kinda curious to hear results from others. I am interested in both routers with and without entware installed.

just do this command

time malware-block

and when the script ends it prints out 3 stats i used the "real" as measurement for my tests so please post avg time and router model and if entware was present.

Around 05:10 for both RT-N66R (single-core MIPS) and RT-AC68U (dual-core ARM).

 

[swetoast]

Tnx everyone for reporting your stats gonna put up a list on load times on the wiki based on all your information so that new users gets a sense on how long the load time takes for the script.

 

[Cedarhillguy]

This is a wonderful script. Have a suggestion for consideration. I noticed when the script updates, it resets the iptables statistics, so you don't know if any devices attempted to reach those blocked IPs.

iptables -vnL FORWARD

Chain FORWARD (Policy DROP 0 packets, 0 bytes)
pkts    bytes  target  Prot opt in out source     destination
    0       0  DROP    ALL  --  *  *   0.0.0.0/0  0.0.0.0/0      match-set  malware-filter src

I'm still a beginner with Linux, but learning by searching the internet. I've modified a script merlin posted for firmware update, so it will send a email if devices do attempt to reach the malware blocked ports between malware updates. It checks the iptables packet count and if it does not = 0, then sends the email.

malware-email

#!/bin/sh

if [ "$(iptables -vnL FORWARD | grep -w 'malware-filter' | cut -c1-5)" != "    0" ]; then

   # SMTP parameters
   SMTP="smtp server address ie smtp.outlook.com"
   PORT="465"
   USERNAME="email account"
   PASSWORD="email password"

   # Mail Enveloppe
   FROM_NAME="Router"
   FROM_ADDRESS="from email address"
   TO_NAME="name"
   TO_ADDRESS="to email address"


   ### Do not change below
   echo "From: \"$FROM_NAME\" <$FROM_ADDRESS>" > /tmp/mail.txt
   echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >> /tmp/mail.txt
   echo "Subject: Malware filter block results" >> /tmp/mail.txt
   echo "" >> /tmp/mail.txt
   iptables -vnL FORWARD 1 >> /tmp/mail.txt

   curl --url smtps://$SMTP:$PORT \
     --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
     --upload-file /tmp/mail.txt \
     --ssl-reqd \
     --user "$USERNAME:$PASSWORD" --insecure

   rm /tmp/mail.txt

fi

It can be scheduled to run separately from the malware-block, however I've added the below line in malware-block; above the first case statement, so it runs just before it updates the ipset.

$path/malware-email

 

[swetoast]

Nice addition! thinking of doing a more in depth page for malware-block and include this script since the community really like this as an additional security feature

 

[tomsk]

Does this work for non entware xargs?.... shame for those folks to miss out.

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 10

path=/opt/var/cache/malware-filter                      # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;

*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
        mkdir -p $path
        wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.pre
        cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-filter.txt
 }

run_ipset () {

get_list

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    cat $path/malware-filter.txt | xargs -P10 -I {} ipset $SYNTAX malware-filter {}
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    cat $path/malware-filter.txt | xargs -P10 -I {} ipset $SYNTAX malware-update {}
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi
}

run_ipset

logger -s -t system "Malware Filter loaded $(cat $path/malware-filter.txt | wc -l) unique ip addresses."
exit $?

 

[swetoast]

no it doesnt and thats nothing i can do anything about so either install entware or do it the old slow way.

 

[tomsk]

no it doesnt and thats nothing i can do anything about so either install entware or do it the old slow way.

Oh well... the busybox xargs must be pretty stripped down..... thats really basic usage

Maybe the busybox version just doesn't like the -P max-procs option. If thats the case we could just omit the -P10 from the xargs line and optionally use that instead of iterating through the for i loop for the s-l-o-o-o-w version ( maybe a "little" faster if it works)

cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-filter {}
Replaces
for i in `cat $path/malware-filter.txt`; do ipset $SYNTAX malware-filter $i ; done
And
cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-update {}
Replaces
for i in `cat $path/malware-filter.txt`; do ipset $SYNTAX malware-update $i ; done

 

[swetoast]

cool ill review and test and if its good ill merge creds as usual

 

[tomsk]

cool ill review and test and if its good ill merge creds as usual

You may want to compare the speed for piping cat through ipset in entware with the -P10 option against the existing version too, it seemed pretty spiffy for me when i tried.

 

[swetoast]

Revision 11 is up

huge tnx to tomsk for finding a way to speed up installations without entware

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 11

path=/opt/var/cache/malware-filter                      # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;

*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
        mkdir -p $path
        wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.pre
        cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-filter.txt
 }

run_ipset () {

get_list

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi
}

run_ipset

logger -s -t system "Malware Filter loaded $(cat $path/malware-filter.txt | wc -l) unique ip addresses."
exit $?

 

[swetoast]

Rev. 12 is up

Changelog: added local lan ip cleaner, too ensure that local lan ips don't get banned

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife
# Testers: shooter40sw
# Revision 12

path=/opt/var/cache/malware-filter                      # Set your path here
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers

    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'

     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;

*v4) # Value for Mips Routers

    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''

    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

get_list () {
        mkdir -p $path
        wget -q --tries=$retries --show-progress -i $path/malware-filter.list -O $path/malware-list.tmp
        awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/'  $path/malware-list.tmp > $path/malware-list.pre
        cat $path/malware-list.pre | grep -oE "$regexp" | sort -u >$path/malware-filter.txt
        if [ -f $path/malware-list.tmp ]; then rm $path/malware-list.tmp; fi
        if [ -f $path/malware-list.pre ]; then rm $path/malware-list.pre; fi
 }

run_ipset () {

get_list

echo "adding ipset rule to firewall this will take time."

ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a $path/malware-filter.txt nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat $path/malware-filter.txt | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi

iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi
}

run_ipset

logger -s -t system "Malware Filter loaded $(cat $path/malware-filter.txt | wc -l) unique ip addresses."
exit $?

 

[myname]

Hello to everybody.
First of all I apologize for my poor english, well a have an RT-AC68U with arm processor at 800Mhz, I have installed merlin 380.65, and after I have installed entware; then I tried to run the script object of this topic.
I cut and paste the version that is in the wiki, but I'm not able to run the script corrctly.
This is the error i get from ssh interface:

./malware-block.sh
ipset v4.5: Kernel ip_set module is of protocol version 6.I'm of protocol version 4.
Please upgrade your kernel and/or ipset(8) utillity.
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: can't insert 'ip_set_nethash.ko': No such file or directory
insmod: can't insert 'ip_set_iphash.ko': No such file or directory
insmod: can't insert 'ipt_set.ko': No such file or directory
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>] 160.31K   363KB/s   in 0.4s   
/opt/var/cache/malware-filter/malware-list.tmp           [ <=>                                                                                                                  ]   2.12K  --.-KB/s   in 0s     
/opt/var/cache/malware-filter/malware-list.tmp           [ <=>                                                                                                                  ]  10.88K  --.-KB/s   in 0.05s 
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  15.20K  94.8KB/s   in 0.2s   
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  27.24K  --.-KB/s   in 0.1s   
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  58.11K  --.-KB/s   in 0.04s 
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  19.27K  --.-KB/s   in 0.001s
adding ipset rule to firewall this will take time.
--set option deprecated, please use --match-set
iptables v1.4.14: Set malware-filter doesn't exist.

Try `iptables -h' or 'iptables --help' for more information.
system: Malware Filter loaded 19831
unique ip addresses.

I understood that the problem is th version of ipset, but I'm not able to correct the problem. Does anybody help me?

 

[tomsk]

Hello to everybody.
First of all I apologize for my poor english, well a have an RT-AC68U with arm processor at 800Mhz, I have installed merlin 380.65, and after I have installed entware; then I tried to run the script object of this topic.
I cut and paste the version that is in the wiki, but I'm not able to run the script corrctly.
This is the error i get from ssh interface:

./malware-block.sh
ipset v4.5: Kernel ip_set module is of protocol version 6.I'm of protocol version 4.
Please upgrade your kernel and/or ipset(8) utillity.
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: can't insert 'ip_set_nethash.ko': No such file or directory
insmod: can't insert 'ip_set_iphash.ko': No such file or directory
insmod: can't insert 'ipt_set.ko': No such file or directory
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>] 160.31K   363KB/s   in 0.4s 
/opt/var/cache/malware-filter/malware-list.tmp           [ <=>                                                                                                                  ]   2.12K  --.-KB/s   in 0s   
/opt/var/cache/malware-filter/malware-list.tmp           [ <=>                                                                                                                  ]  10.88K  --.-KB/s   in 0.05s
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  15.20K  94.8KB/s   in 0.2s 
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  27.24K  --.-KB/s   in 0.1s 
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  58.11K  --.-KB/s   in 0.04s
/opt/var/cache/malware-filter/malware-list.tmp       100%[=====================================================================================================================>]  19.27K  --.-KB/s   in 0.001s
adding ipset rule to firewall this will take time.
--set option deprecated, please use --match-set
iptables v1.4.14: Set malware-filter doesn't exist.

Try `iptables -h' or 'iptables --help' for more information.
system: Malware Filter loaded 19831
unique ip addresses.

I understood that the problem is th version of ipset, but I'm not able to correct the problem. Does anybody help me?

AC68U on 380.65 should be using ipset V6.... if you just type ipset at the command line what does it say?

And just to be sure of the script.... what is the version number from the wiki...you may have an old version from somewhere.

 

[myname]

Thaks for your answer, the vesionof ipset is

admin@RT-AC68U:/tmp/home/root# ipset
ipset v4.5: no command specified
Try `ipset -H' or 'ipset --help' for more information.

And the versione of the script from the wiki is revision 13

 

[myname]

An other information, if give the command

admin@RT-AC68U:/tmp/mnt/sda1/entware-ng.arm/bin# ipset -v
ipset v4.5, protocol version 4.
ipset v4.5: Kernel ip_set module is of protocol version 6.I'm of protocol version 4.
Please upgrade your kernel and/or ipset(8) utillity.

this is the result

 

[tomsk]

Something a bit strange going on here.... the malware filter script is trying to load the ipset modules for version 4 because it is testing for it in the same way. However i believe Merlin has built 380.65 with V6 . You may have to do a factory reset to clear all your old nvram values if you have come from a much older build.

Im on the same hardware/firmware as you but....

ipset v6.29, protocol version: 6