Malware Filter / bad host IPSET

[swetoast]

/tmp/mnt/sda1/malware-filter# tail -f /tmp/syslog.log
Feb 20 01:00:00 crond[247]: USER pid 7886 cmd /tmp/mnt/sda1/malware-filter/malware-block
Feb 20 01:00:05 system: Malware Filter loaded 0 unique ip addresses.

So are you storing your malware-filter.list in the same dir and is the path set to that file in the blocklist path ?

 

[shooter40sw]

So are you storing your malware-filter.list in the same dir and is the path set to that file in the blocklist path ?

Yes

 

[swetoast]

and it works when you run it manually? do you have entware installed ?

and last but not least mind printing the value of this echo $TMP

 

[shooter40sw]

Yes I got entware installed, it works perfect if I run it manually

/root# echo $TMP
/opt/tmp

and it works when you run it manually? do you have entware installed ?

and last but not least mind printing the value of this echo $TMP

 

[swetoast]

ill investigate during the weekend until then run an older revision cause i think there is a bug present

 

[shooter40sw]

good morning, just to confirm this is the result in the syslog of the manual run

eb 20 14:46:32 kernel: net/ipv4/netfilter/ip_set_iphash.c: iphash_retry: rehashing of set malware-update triggered: hashsize grows from 39366 to 59049
Feb 20 14:46:59 system: Malware Filter loaded 18433 unique ip addresses.

and now the cron run

Feb 21 00:00:00 crond[247]: USER xxx pid 6089 cmd /tmp/mnt/sda1/malware-filter/malware-block
Feb 21 00:00:06 system: Malware Filter loaded 0 unique ip addresses.

So you can take a look when you have time, thanks

 

[swetoast]

again revert to older version ill look at it when i have time.

 

[swetoast]

Hey everyone havent had time this weekend to fix the script still recommending using revision 15 until the issue is fixed. Just wanted to update everyone so there is no misunderstandings.

 

[tomsk]

Hey everyone havent had time this weekend to fix the script still recommending using revision 15 until the issue is fixed. Just wanted to update everyone so there is no misunderstandings.

4. One rare problem with xargs is end of file string, by default end of file string is "_" and if this string occurs in input the rest of input is ignored by xargs. Though you can change end of file string by using option "-eof". <---do you think your new file naming convention may be somehow tripping the xargs up when the script is run from the cron rather than directly?

 

[swetoast]

give me the one liner fix in pm and ill look at it and update if it works, haven't had a lot of time to fix issues like this when i started my new job so im more then greatful for patches

 

[tomsk]

give me the one liner fix in pm and ill look at it and update if it works, haven't had a lot of time to fix issues like this when i started my new job so im more then greatful for patches

Just a wild guess actually.... i haven't copied his setup to try and reproduce his bug.... @shooter40sw might want to try it himself if you dont have time..... just change the underscores in the file names so
"$TMP/malware_filter_sorted.part" becomes "$TMP/malware-filter-sorted.part" for example.

the cleanup can be fixed later if it works

If not it has to be some weird shell variable problem with $TMP... i have a feeling a job run from cron doesn't have the same environment

 

[swetoast]

thinking if i hardcode /tmp instead of variable $tmp it might work better, made the changes in git.

 

[visortgw]

thinking if i hardcode /tmp instead of variable $tmp it might work better, made the changes in git.

I experienced the same issue with the cron job not working. It appears that both changing $TMP to /tmp and changing the underscores to dashes are required to resolve the issue.

 

[swetoast]

kewl, if you got the time please review my git (to other users please dont use this version its WIP), if it all works then ill update as soon as ive tested or if someone confirms the review

 

[swetoast]

There's no single answer to this, as it depends on the type of WAN, also whether you use dual wan or not.

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/shared/rtstate.c#L155

So it can be wan0_ifname, or wan0_pppoe_ifname. And if you use dual WAN, then unit 1 would be wan1_*.

so RMerlin, if i was to use regexp to match those values would this be good enough ?

^wan[0-1]_(ifname|pppoe_ifname|pptp_ifname|l2tp_ifname|gw_ifname)$

 

[visortgw]

kewl, if you got the time please review my git (to other users please dont use this version its WIP), if it all works then ill update as soon as ive tested or if someone confirms the review

Malware filter in git correctly runs from cron job as well as from command line on both mips and arm routers.

 

[swetoast]

Rev 17, sorry for the delay

#!/bin/sh
# Author: Toast
# Contributers: Octopus, Tomsk, Neurophile, jimf, spalife, visortgw
# Testers: shooter40sw
# Revision 17

blocklist=/jffs/malware-filter.list                     # Set your path here
failover=eth0                                           # Change only if WAN interface is not detected.
retries=3                                               # Set number of tries here
regexp=`echo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"`         # Dont change this value

case $(ipset -v | grep -oE "ipset v[0-9]") in
*v6) # Value for ARM Routers
    MATCH_SET='--match-set'
    HASH='hash:ip'
    SYNTAX='add'
    SWAPPED='swap'
    DESTROYED='destroy'
    OPTIONAL='family inet hashsize 2048 maxelem 65536'
     ipsetv=6
     lsmod | grep "xt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set
     do
          insmod $module
     done
;;
*v4) # Value for Mips Routers
    MATCH_SET='--set'
    HASH='iphash'
    SYNTAX='-q -A'
    SWAPPED='-W'
    DESTROYED='--destroy'
    OPTIONAL=''
    ipsetv=4
     lsmod | grep "ipt_set" > /dev/null 2>&1 || \
     for module in ip_set ip_set_nethash ip_set_iphash ipt_set
     do
          insmod $module
     done
;;
esac

check_online () {
if [ -z "$(which nvram)" ]; then
iface=`grep "$failover" /proc/net/dev`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
     then get_list; fi
     else exit 1; fi
else iface=`nvram get wan0_ifname`
if   [ -n "$iface" ]; then
     if [ $(curl -s https://4.ifcfg.me/ | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b") ]
     then get_list; fi
     else exit 1; fi
fi }

get_list () {
url=https://gitlab.com/swe_toast/malware-filter/raw/master/malware-filter.list
if [ ! -f $blocklist ]
then wget $url -O $blocklist; get_source; else get_source; fi
}

get_source () {
    wget -q --tries=$retries --show-progress -i $blocklist -O /tmp/malware-filter-raw.part
        awk '!/(^127\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/' /tmp/malware-filter-raw.part > /tmp/malware-filter-presort.part
    cat /tmp/malware-filter-presort.part | grep -oE "$regexp" | sort -u > /tmp/malware-filter-sorted.part
}

run_ipset () {
echo "adding ipset rule to firewall this will take time."
ipset -L malware-filter >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$(ipset --swap malware-filter malware-filter 2>&1 | grep -E 'Unknown set|The set with the given name does not exist')" != "" ]; then
    nice -n 2 ipset -N malware-filter $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-filter PARAM
    else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-filter {}; fi
fi
else
    nice -n 2 ipset -N malware-update $HASH $OPTIONAL
    if [ -f /opt/bin/xargs ]; then
    /opt/bin/xargs -P10 -I "PARAM" -n1 -a /tmp/malware-filter-sorted.part nice -n 2 ipset $SYNTAX malware-update PARAM
    else cat /tmp/malware-filter-sorted.part | xargs -I {} ipset $SYNTAX malware-update {}; fi
    nice -n 2 ipset $SWAPPED malware-update malware-filter
    nice -n 2 ipset $DESTROYED malware-update
fi
iptables -L | grep malware-filter > /dev/null 2>&1
if [ $? -ne 0 ]; then
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
else
    nice -n 2 iptables -D FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
    nice -n 2 iptables -I FORWARD -m set $MATCH_SET malware-filter src,dst -j REJECT
fi }

cleanup () {
logger -s -t system "Malware Filter loaded $(cat /tmp/malware-filter-sorted.part | wc -l) unique ip addresses."
find /tmp -type f -name 'malware-filter-*.part' -delete
}

check_online
run_ipset
cleanup

exit $?

 

[swetoast]

just outta curiosity how much is your malware filter catching ?

ipset -L malware-filter | wc -l

i got 7 lists in my filter and my block is ~49K of blocked IP to be specific 49815

just remember that its minus 7 of the given value since its seven lines of text that gets counted into the wc -l command

 

[steelskinz]

/jffs/scripts# ipset -L malware-filter | wc -l
ipset v6.29: The set with the given name does not exist
0

 

[swetoast]

clearly not loaded correctly, rerun malware-filter at the latest revision.