Yet another malware block script using ipset (v4 and v6)

[thelonelycoder]

Oops, sorry about that . Need to add a note to AB users to whitelist that domain

That does not help if your script blocks the IP.
AB works fine, dnsmasq resolves the domain but one of your rules blocks the IP.

 

[steelskinz]

Is the max IP blocked is 65535 ? I guess that's wjy there is 3 sets for IP ?

 

[redhat27]

That does not help if your script blocks the IP.
AB works fine, dnsmasq resolves the domain but one of your rules blocks the IP.

I think you may have misunderstood. I meant that users of this script (who are also AB Solution users) will need to whitelist the pgl.yoyo.org domain. Since ya-malware-block does not have its own whitelisting, but there is another script that I maintain that do, the way to whitelist a domain is to add that to the whitelist domains file there, and then run both the scripts together: First the ya-malware-block and then the iblocklist-loader (to effect the whitelisting). I have tried this out myself, and I'm able to update the hosts lists using your AB script.

 

[redhat27]

Is the max IP blocked is 65535 ? I guess that's wjy there is 3 sets for IP ?

That is a hard limitation of ipset v4 only. Since this script does both v4 and v6, the handling is done the same way for both versions.

 

[redhat27]

Added a whitelisting section on post #1

 

[LUZIFER]

iblocklist-loader.sh for Whitelisting, ok.

Many code for anything else, I do not need it all.
What do I need in this script for Whitelisting only ?

something like "whitelisting-loader.sh"

 

[redhat27]

I was trying to keep this script short and to the point. You'd typically not want to whitelist malware. That whitelisting in the OP example is only needed in special cases where unfortunately the pgl.yoyo.org was (mistakenly?) included in the Firehol lists (you would want to whitelist this if you are using AB Solution for adblocking)

You should not need to enable any source in the iblocklist-loader script. Use the shorter version, and then have

BLOCKLIST_INDEXES=""

as defined here. Have a /jffs/ipset_lists/whitelist-domains.txt file with an entry like this, and modify your cru like mentioned in the OP. You'll be set. This iblocklist-loader will not load anything else other than the whitelist

 

[e2301010]

Add to the end of ya-malware-block.sh and pgl.yoyo.org should resolve and will be part of the schedule . Now give me some likes. JK!

WISH LIST: YAMB integrated with AB-Solution

ipset -D YAMalwareBlock1IP 213.230.210.230

 

[thelonelycoder]

Add to the end of ya-malware-block.sh and pgl.yoyo.org should resolve and will be part of the schedule . Now give me some likes. JK!

WISH LIST: YAMB integrated with AB-Solution

ipset -D YAMalwareBlock1IP 213.230.210.230

Not going to happen by me, these are entirely different approaches to blocking.
One is from the inside out, the other blocks from the outside in (mostly).

 

[redhat27]

Add to the end of ya-malware-block.sh and pgl.yoyo.org should resolve and will be part of the schedule . Now give me some likes. JK!

WISH LIST: YAMB integrated with AB-Solution

ipset -D YAMalwareBlock1IP 213.230.210.230

You can do it if you want, but it is not definite in which set exactly it will go (YAMalwareBlock1IP or YAMalwareBlock2IP or YAMalwareBlock3IP)
Also, there should be a general way of adding the exceptions, not on the scripts themselves.

 

[redhat27]

I modified the OP to indicate that you do not actually need to put the whitelisting in cron. Just once in services start will do: That is because this script will not re-create the iptable rules if they are already existing. It will only refresh the ipset (malware sources) data

 

[e2301010]

I know. The best way to do this would be to have an external whitelist txt file that a user can enter an IP per line.

Then at the end of your script, do a loop test for each IP in the whitelist.txt if exists in each of the sets

ipset -T setname1 ipaddress1
ipset -T setname2 ipaddress1
ipset -T setname3 ipaddress1

If exists then proceed to delete from set

ipset -D setnameX ipaddres1

BTW: great work on this script. It's very fast!

You can do it if you want, but it is not definite in which set exactly it will go (YAMalwareBlock1IP or YAMalwareBlock2IP or YAMalwareBlock3IP)
Also, there should be a general way of adding the exceptions, not on the scripts themselves.

 

[redhat27]

I know. The best way to do this would be to have an external whitelist txt file that a user can enter an IP per line.

Then at the end of your script, do a loop test for each IP in the whitelist.txt if exists in each of the sets

See posts #167 and #171. If there is an established way to do it already, wouldn't that be just extra processing? I'd like to keep this as small and lean as possible. Having said that, you can obviously modify this to your liking

BTW: great work on this script. It's very fast!

Thank you. I have an even faster one coming up shortly. Cuts the runtime by more than half. I'll post here soon (maybe tomorrow)

 

[redhat27]

@e2301010 I take back what I said just earlier. I will implement a whitelist in this script in the next version

 

[Xentrk]

Dropbox syncing broken too

Sorry for the late reply. I can confirm that my dropbox sync is working fine.

 

[shooter40sw]

Hi @redhat27 Im using ab solutions from @thelonelycoder and I dont see to be able to whitelist pgl.yoyo.org after whitelisting it, Im using the ipset script running at boot in the post-mount script, and then a manually run your other malware script, Im not clear how to make this work, please help, thanks

 

[Xentrk]

Hi @redhat27 Im using ab solutions from @thelonelycoder and I dont see to be able to whitelist pgl.yoyo.org after whitelisting it, Im using the ipset script running at boot in the post-mount script, and then a manually run your other malware script, Im not clear how to make this work, please help, thanks

Same issue here as well. I have this in services-start

sh /jffs/scripts/create-ipset-lists.sh
sh /jffs/scripts/IPSET_Block.sh init ipset
sh /jffs/scripts/ya-malware-block.sh
sh /jffs/scripts/iblocklist-loader.sh

/jffs/ipset_lists/whitelist-domains.txt

apple.com # blocked by TBG Primary-Threats
bbc.co.uk # blocked by TBG Primary-Threats
eotugame.com #blocked by Squidblacklist Malicious (premium)
icloud.com # blocked by TBG Primary-Threats
mail.live.com # Hotmail blocked by TBG Primary-Threats
speedtest.net # blocked by Yoyo Adservers
teamviewer.com # blocked by Bluetack Level1
bing.com # bing search engine
pgl.yoyo.org # blocked by FireHOL lists used by ya-malware-block

 

[redhat27]

Version 2.0 is out!

• This version brings major speed improvements by using the already curated FireHOL lists aggregation.
• No more empty set YAMalwareBlock3IP. If you enable FireHOL Level4, you will get automatically get YAMalwareBlock3IP, but not empty.
• Also, if you add more custom sources, YAMalwareBlock3IP, YAMalwareBlock4IP, YAMalwareBlock5IP etc will be automatically created at 64k increments
• IP whitelisting. The IP for pgl.yoyo.org (AB-Solution hosts source) is automatically whitelisted by default. You can add custom whitelisted IPs too

Note:
This script now uses 2 external files: ya-malware-block.urls (formerly ya-malware-block.url_list) and a new file ya-malware-block.whites. Both will be downloaded from github unless they already exist on your router.

Please delete the old /jffs/ipset_lists/ya-malware-block.url_list, it is not used anymore:

rm /jffs/ipset_lists/ya-malware-block.url_list

I will update the post #1 shortly

 

[redhat27]

@shooter40sw @Xentrk @e2301010 @LUZIFER Please use the latest version: pgl.yoyo.org is automatically whitelisted. You will not need to use the domain whitelisting feature of iblocklist-loader script for this anymore.

 

[LUZIFER]

running

/jffs/scripts/ya-malware-block.sh

then

/jffs/scripts$ /jffs/scripts/ya-malware-block.sh
ipset v6.29: Error in line 13899: Element cannot be added to the set: it's already added
ipset v6.29: Error in line 4663: Element cannot be added to the set: it's already added

May 20 18:00:01 Firewall: /jffs/scripts/ya-malware-block.sh: Adding ya-malware-block rules to firewall...
May 20 18:00:06 Firewall: /jffs/scripts/ya-malware-block.sh: Loaded sets YAMalwareBlock1IP (0) YAMalwareBlock2IP (5975) and YAMalwareBlockCIDR (0) in 5 seconds

Something is not working yet