Malware Filter / bad host IPSET

[spalife]

@Nutz2U2 Meeh peerguarian is pretty useless imho but i can take a look at it if its important too ya
@spalife i knew i missed something, nice catch !! shirt happens when you script when you got the flu gonna take a look at it when im feeling better

Snow Time...Santa Time...Flu Time

Hope you are feeling better...thanks for the script.

I only made two changes for it to get working on my N66U
1. Added option --no-check-certificate for the wget command for the https cert issues

2. Reverted to grep 'Unknown set' as N66U has a ipset 4.5 version

 

[swetoast]

i could make a better grep if you want so you dont have to change future versions as for wget i could look and see if that can be resolved since merlin includes certs if you dont have entware installed but what i need from you is too just post that line

 

[spalife]

i could make a better grep if you want so you dont have to change future versions as for wget i could look and see if that can be resolved since merlin includes certs if you dont have entware installed but what i need from you is too just post that line

I do have entware-ng installed and am on 380.63_2 on N66U.
Running the malware filter script from an attached USB and refreshing
it bi-weekly.

1)
wget complaints about not able to resolve https certs of the block list sites,
so the need to include --no-check-certificate option.

2)
It's a small change in message from version difference of ipset 4.5 to ipset 6.
The ipset 4.5 message is
grep 'Unknown set'
&
ipset 6 message is
grep -E 'Unknown set|The set with the given name does not exist'

3)
Had to include the below in firewall-start to compensate the iptables flush
whenever the router is restarted or services are restarted.
iptables -I FORWARD -m set --set malware-filter src,dst -j REJECT

4)
All other block list sites work, except
http://cinsscore.com/list/ci-badguys.txt
Removed it from malware-filter.list file as it holds up the download.

After the above minor changes, everything is working fine.

 

[RMerlin]

wget complaints about not able to resolve https certs of the block list sites,
so the need to include --no-check-certificate option.

Make sure you use the wget applet that comes with the firmware. I added a root certificate catalog quite a while ago, so wget can properly validate certificates.

 

[spalife]

Indeed.
There seems to be a conflict with entware-ng and firmware
when it comes to wget.

As I have entware-ng the scripts are defaulting to
/opt/bin/wget which is of version 1.18

and the firmware /usr/sbin/wget which is of version 1.16

Quick Q in relation to the firmware wget,
do we still need to do the below in /jffs/scripts/services-start ?

echo "ca-certificate=/rom/ca-bundle.crt" > ~/.wgetrc

The above was mentioned @
http://www.snbforums.com/threads/certificate-verification-issue-with-rt-n66u.25248/

 

[spalife]

Q clarified by
If WGETRC is not set, Wget will try to load $HOME/.wgetrc.

Reference:
https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Location.html#Wgetrc-Location

 

[RMerlin]

The firmware has its own wgetrc file.

lrwxrwxrwx    1 admin    root            15 Dec 31  1969 /etc/wgetrc -> /rom/etc/wgetrc

 

[spalife]

The firmware has its own wgetrc file.

lrwxrwxrwx    1 admin    root            15 Dec 31  1969 /etc/wgetrc -> /rom/etc/wgetrc

I have seen that but since it is not in the locations looked at runtime by wget, the ca-certificates bundle packaged with the firmware does not get used
unless a manual option is specified through --ca-certificate.

Hence, my previous issue of https certificate failures with both entware-ng as well as firmware supplied wget.

Once a user specified .wgetrc file (see ref below) is located at $HOME/.wgetrc, I had no issues with ca_certificate bundle with firmware and as well as entware-ng wget.

https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Location.html#Wgetrc-Location

 

[RMerlin]

I have seen that but since it is not in the locations looked at runtime by wget, the ca-certificates bundle packaged with the firmware does not get used
unless a manual option is specified through --ca-certificate.

Wget looks at /etc/wgetrc at run time - look it up on its manpage.

FILES
/etc/wgetrc
Default location of the global startup file.

.wgetrc
User startup file.

Again, SSL works just fine with the built-in wget binary:

admin@Stargate88:/tmp/home/root# wget https://www.google.ca
--2016-12-30 23:36:04--  https://www.google.ca/
Resolving www.google.ca... 2607:f8b0:400b:807::2003, 206.248.151.95, 206.248.151.117, ...
Connecting to www.google.ca|2607:f8b0:400b:807::2003|:443... failed: Network is unreachable.
Connecting to www.google.ca|206.248.151.95|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'

index.html                                      [ <=>                                                                                        ]  11.14K  --.-KB/s   in 0.002s 

2016-12-30 23:36:04 (6.79 MB/s) - 'index.html' saved [11404]

admin@Stargate88:/tmp/home/root#

 

[Xentrk]

@swetoast

Not sure if I did the installation of the malware filter correctly. I copied the code from github and saved it in /jffs/scripts/malware-filter. When I ran the script, I got the following error message which I see is probably normal from prior posts:

insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists

The next step I did was to create the malware-filter.list file in /opt/var/cache/malware-filter

I am able to get ping responses from some of the IP addresses in the malware-filter.list but not all, which makes me believe I did something wrong in the install or it is taking the router awhile to build the chain rules. Do I need to have an entry in my firewall-start file perhaps? Here are commands for help in debugging. Thanks in advance for the help!

admin@RT-AC88U-5248:/jffs/scripts# ipset --l
Name: malware-filter
Type: hash:ip
Revision: 0
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 8248
References: 1
Members:

admin@RT-AC88U-5248:/jffs/scripts# iptables -L | grep malware
REJECT all -- anywhere anywhere match-set malware-filter src,dst reject-with icmp-port-unreachable

admin@RT-AC88U-5248:/jffs/scripts# iptables –L
Chain FORWARD (policy DROP)
target prot opt source destination
REJECT all -- anywhere anywhere match-set malware-filter src,dst reject-with icmp-port-unreachable
TCPMSS tcp -- anywhere anywhere tcpflags: SYN,RST/SYN TCPMSS clamp to PMTU
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere state INVALID
ACCEPT all -- anywhere anywhere
SECURITY all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate DNAT
ACCEPT all -- anywhere anywhere

admin@RT-AC88U-5248:/jffs/scripts# /usr/sbin/iptables -L FORWARD -v –n
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 match-set malware-filter src,dst reject-with icmp-port-unreachable
19036 1112K TCPMSS tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcpflags: 0x06/0x02 TCPMSS clamp to PMTU
12M 16G ACCEPT all -- tun11 * 0.0.0.0/0 0.0.0.0/0
7302K 941M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 DROP all -- !br0 ppp0 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- !br0 eth0 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state INVALID
2 104 ACCEPT all -- br0 br0 0.0.0.0/0 0.0.0.0/0
0 0 SECURITY all -- ppp0 * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate DNAT
14270 919K ACCEPT all -- br0 * 0.0.0.0/0 0.0.0.0/0

Here is the output of ipset script in firewall-start:

admin@RT-AC88U-5248:/jffs/scripts# . ./firewall-start
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: 'ip_set_nethash.ko': module not found
insmod: 'ip_set_iphash.ko': module not found
insmod: 'ipt_set.ko': module not found
iptables v1.4.14: Set TorNodes doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: Set BlockedCountries doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: Set MicrosoftSpyServers doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.

 

[Xentrk]

No content in malware-filter.txt file. I created malware-filter.list after I ran the script.

 

[Xentrk]

No files in /jffs/ipset_lists/

 

[swetoast]

Seems like your paths are off make sure they are pointing towards the same place so the script can run proper

 

[tomsk]

does the malware/bad host filter play well with the trend micro Aiprotection stuff turned on and will it provide any extra protection or is it just a duplication of effort?

 

[swetoast]

well since trend micros stuff is proprietary, its hard to say what it does mine is much easier to follow. i think of it as an extra layer and yes it plays well with aiprotection

 

[Xentrk]

Seems like your paths are off make sure they are pointing towards the same place so the script can run proper

Thank you @swetoast! I noticed earlier today when I attempted to install that I did not have an /opt/var directory. Rather a symbolic link of opt -> /tmp/opt that points to /tmp/mnt/entware/entware/ that may have happened during install of entware or ab solution 3. That gives me something to work with and I will pick it up tomorrow.

 

[steelskinz]

Great script. Thanks !

 

[tomsk]

Here is the output of ipset script in firewall-start:

admin@RT-AC88U-5248:/jffs/scripts# . ./firewall-start
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
insmod: 'ip_set_nethash.ko': module not found
insmod: 'ip_set_iphash.ko': module not found
insmod: 'ipt_set.ko': module not found
iptables v1.4.14: Set TorNodes doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: Set BlockedCountries doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: Set MicrosoftSpyServers doesn't exist.
Try `iptables -h' or 'iptables --help' for more information.

If you an an AC88U with 380.64 aren't these are the wrong modules?.... these look like v4 and not v6
The first script in the ipset wiki hasn't been modified to reflect the changes to detect which ipset...

 

[Xentrk]

If you an an AC88U with 380.64 aren't these are the wrong modules?.... these look like v4 and not v6
The first script in the ipset wiki hasn't been modified to reflect the changes to detect which ipset...

Thank you for pointing that out @tomsk! I am learning something new everyday. I will focus on the malware-filter script for now.

 

[Xentrk]

Thank you @swetoast! I noticed earlier today when I attempted to install that I did not have an /opt/var directory. Rather a symbolic link of opt -> /tmp/opt that points to /tmp/mnt/entware/entware/ that may have happened during install of entware or ab solution 3. That gives me something to work with and I will pick it up tomorrow.

@swetoast The malware filter appears to have installed correctly. I did not correct anything with my path. I only reran the script. But like I said previously, I did not install the malware-filter.list file until after I ran the script in my first install attempt. Perhaps it is a prerequisite to have the malware-filter.list file created first?

admin@RT-AC88U-5248:/jffs/scripts# ./malware-filter
insmod: can't insert '/lib/modules/2.6.36.4brcmarm/kernel/net/netfilter/ipset/ip_set.ko': File exists
/opt/var/cache/malw 100%[====================>] 159.95K 145KB/s in 1.1s
/opt/var/cache/malw [ <=> ] 2.34K --.-KB/s in 0s
/opt/var/cache/malw [ <=> ] 10.51K 28.2KB/s in 0.4s
/opt/var/cache/malw 100%[====================>] 17.00K 59.0KB/s in 0.3s
/opt/var/cache/malw 100%[====================>] 278.52K 152KB/s in 1.8s
/opt/var/cache/malw 100%[====================>] 6.11K 20.7KB/s in 0.3s

I see ip addresses are now populated in malware-filter.txt and malware-list.pre files. But, I can get ping replies from some of the ip addresses which makes me believe something still not correct. Here are two for example.

1.221.200.251, 100.6.61.161

Is there an entry required in services-start or firewall-start? Thank you again for the help!

edit, just saw this in post #43:

3)
Had to include the below in firewall-start to compensate the iptables flush
whenever the router is restarted or services are restarted.
iptables -I FORWARD -m set --set malware-filter src,dst -j REJECT

After the above minor changes, everything is working fine.

edit 2: received an error msg when trying to run the above example in firewall-start:
--set option deprecated, please use --match-set
I changed to the following and it works:
#!/bin/sh
iptables -I FORWARD -m set --match-set malware-filter src,dst -j REJECT