Menu
What's new
By:
Filters Better Search

Skynet Filter Validator v0.7 - Skynet Firewall Filter List IPv4 Integrity Validator

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Viktor Jaep

Viktor Jaep

Part of the Furniture
Filter Validator v0.7
A collaborative project between @Viktor Jaep and @SomeWhereOverTheRainBow!

Filter Validator tests the IPv4 addresses (and IPv6 if present) on a given filter list that are to be used with the Skynet Firewall running on Asus-Merlin Firmware in order to block incoming/outgoing IPs. This script arose out of the need to determine exactly which blacklist URL contained an invalid IP that was causing our Skynet firewalls to fail importing the correct IP sets due to an invalid IP somewhere on these lists.

**EDIT** -- as of March 10, 2023 -- Skynet v7.3.6 now has the necessary regex fixes included to filter out invalid addresses that were breaking the script, wholly inspired by Filter Validator, with many thanks to @SomeWhereOverTheRainBow for his excellent contributions making both scripts even better! Filter Validator may still play a role to determine if blacklist operators are continuing to maintain IP address data integrity, but also giving you a sense on how many entries are in your filter lists. Skynet has a hard limit of 500,000 entries at this point, and Filter Validator may tell you if you're getting too close. Last, other blacklists may stop being supported, or even disappear -- and while Skynet wouldn't give you any indication of this happening, Filter Validator will.

1678533684338.png


If there's an issue with one of the lists itself, or an IP on any of these lists, you'll see something like this:

1678536033496.png


After it's completed validating each entry, it will give you a summary at the end:

1678536120354.png


Usage Guide​

Execute the script as such: sh /jffs/scripts/filtervalidator.sh

Upon execution, it will ask for a valid URL to the specified filter list to be tested. For example, here is are a few valid filter list URLs that will be used if you press enter:

https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

NOTE: Should any list come back with any invalid IP entries (marked in Red), it would be advisable to remove or #COMMENT out the offending entry in your filter list in order to get Skynet back in working condition, or get in touch with the entity that takes care of the list in order to correct their mistake.

Download​

Code: 
curl --retry 3 "https://raw.githubusercontent.com/ViktorJp/FilterValidator/master/filtervalidator.sh" -o "/jffs/scripts/filtervalidator.sh" && chmod 755 "/jffs/scripts/filtervalidator.sh"
 
Last edited:
<RESERVED>
 
Decided to just create it's own thread instead of keeping it buried under another thread. ;) I'm sure we'll be seeing some worthy additions to this soon! Thanks for your collaborative partnership on this, @SomeWhereOverTheRainBow!
 
Last edited:
Last edited:
dave14305 said:
Who is submitting a pull request to @Adamm to catch these malformed IPs during banmalware? He won’t bite (I think).
Click to expand...

This is all news to me. Is the regex broken? o_O
 
Adamm said:
This is all news to me. Is the regex broken? o_O
Click to expand...
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
 
Just saying... @SomeWhereOverTheRainBow is a master at optimizing regex... just published v0.3 that runs the more optimized code combining the validation of both IPv4 and IPv6 addresses. :)
 
dave14305 said:
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
Click to expand...
Don't forget about octal formats like 01.01.01.01 if those are present in the list the regex skynet uses will pull those in as well.
 
dave14305 said:
It doesn’t catch invalid IPs like recent examples 451.91.115.161, 447.96.132.96 or 198.265.75.69 which seem to slip into lists from threatview.io when users make custom filter lists.

ipset restore fails on the invalid IPs.
Click to expand...
And some lists mix IPv6 addressing into their IPv4 lists as well... fun stuff. ;)
 
grep: warning: stray \ before #
 
Viktor Jaep said:
? What's up, @visortgw?
Click to expand...
Sorry about the abrupt message — I was in a hurry. This error is displayed when running v0.3 for every file using your filter list even though every file reports valid.
 
visortgw said:
Sorry about the abrupt message — I was in a hurry. This error is displayed when running v0.3 for every file using your filter list even though every file reports valid.
Click to expand...
Could you please elaborate on the error? Also, could you please provide me the URL for the list you're trying to validate?
 
Viktor Jaep said:
Could you please elaborate on the error? Also, could you please provide me the URL for the list you're trying to validate?
Click to expand...
It’s from /opt/bin/grep versus /bin/grep.
 
Viktor Jaep said:
Could you please elaborate on the error? Also, could you please provide me the URL for the list you're trying to validate?
Click to expand...
Code: 
   _____ ____            _   __     ___    __     __         
  / __(_) / /____ ____  | | / /__ _/ (_)__/ /__ _/ /____  ____
 / _// / / __/ -_) __/  | |/ / _ '/ / / _  / _ '/ __/ _ \/ __/
/_/ /_/_/\__/\__/_/     |___/\_,_/_/_/\_,_/\_,_/\__/\___/_/   v0.3
        By @Viktor Jaep and @SomewhereOverTheRainbow

Filter Validator was designed to run through your Skynet filter lists to determine
if all IP addresses fall within their normal ranges. Should any entries not follow
standard IP rules, they will be identified below. NOTE: Having invalid IPs within
these filter sets will cause the Skynet firewall to malfunction due to regex issues
that are not filtering out bad IPs, causing a loss of blocked IPs and ranges to occur.

Please enter a valid filter list URL, or hit <ENTER> to use example below:
Example 1: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
Example 2: https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

URL: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list

Testing against: https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list

[Downloading Filter List]...OK
[Checking Filter List Contents]...OK

grep: warning: stray \ before #
Checking https://feodotracker.abuse.ch/downloads/ipblocklist.txt
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/alienvault_reputation.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/bds_atif.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/bi_any_2_30d.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/blocklist_net_ua.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/coinbl_hosts_browser.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/cybercrime.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dm_tor.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dshield.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/dyndns_ponmocup.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_block.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_compromised.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_spamhaus.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/et_tor.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level1.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level2.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/firehol_level3.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/greensnow.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/ciarmy.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/iblocklist_ciarmy_malicious.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/iblocklist_pedophiles.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/malc0de.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/maxmind_proxy_fraud.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/normshield_high_attack.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/normshield_high_bruteforce.ipset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/spamhaus_drop.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/spamhaus_edrop.netset
[Valid]

grep: warning: stray \ before #
Checking https://iplists.firehol.org/files/urlvir.ipset
[Valid]

grep: warning: stray \ before #
Checking https://sigs.interserver.net/iprbl.txt
[Valid]

grep: warning: stray \ before #
Checking https://raw.githubusercontent.com/SecOps-Institute/Tor-IP-Addresses/master/tor-exit-nodes.lst
[Valid]

grep: warning: stray \ before #
Checking https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt
[Valid]

grep: warning: stray \ before #
Checking https://www.talosintelligence.com/documents/ip-blacklist
[Valid]

grep: warning: stray \ before #
Checking https://voipbl.org/update
[Valid]
 
dave14305 said:
It’s from /opt/bin/grep versus /bin/grep.
Click to expand...
Maybe I'm missing something... but I'm not seeing any screenshot or other info to come to that conclusion... but I bet you're right. ;)
 
Viktor Jaep said:
Maybe I'm missing something... but I'm not seeing any screenshot or other info to come to that conclusion... but I bet you're right. ;)
Click to expand...
It is caused from this line of code

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -v '^\s*$\|^\s*\#' | sed -n $listcount'p') 2>&1

try

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -vE '^[[:space:]]*#' | sed -n $listcount'p') 2>&1
 
Last edited:
SomeWhereOverTheRainBow said:
It is caused from this line of code

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -v '^\s*$\|^\s*\#' | sed -n $listcount'p') 2>&1

try

blacklisturl=$(cat /jffs/scripts/filter.txt | grep -vE '^[[:space:]]*#' | sed -n $listcount'p') 2>&1
Click to expand...
@SomeWhereOverTheRainBow That fixes it! Thanks! (It appears to run much more quickly, too!)
 
You must log in or register to reply here.

Similar threads

W
Looking for an add-on that does...
Replies
5
Views
2K
JGrana
JGrana
Klonoa
Skynet Skynet causes router to crash and reboot
Replies
1
Views
738
dave14305
dave14305
R
Skynet ASN Banning
Replies
3
Views
1K
SomeWhereOverTheRainBow
SomeWhereOverTheRainBow
Mogsy
Skynet Skynet WebUI as scMerlin sitemap
Replies
7
Views
2K
SomeWhereOverTheRainBow
SomeWhereOverTheRainBow
MWGlidden
Skynet Curl "no URL specified" error when changing Skynet blacklist
Replies
2
Views
1K
djtech2k
D
Similar threads
Thread starter Title Forum Replies Date
thelonelycoder Diversion Need help with selecting new set of filter lists / block lists for Diversion 5.0 Asuswrt-Merlin AddOns 21
L Skynet I have installed the skynet firewall how do I configure it if the security of iot devices is important? Asuswrt-Merlin AddOns 2
Davidncali001 Skynet Message on SKYNET I havent seen before Asuswrt-Merlin AddOns 1
S Skynet firewall reduces wireless range Asuswrt-Merlin AddOns 14
P Skynet Internet speed lower with Skynet on Asuswrt-Merlin AddOns 9
N Skynet SkyNet/Firewall blocking all ping requests, including outbound Asuswrt-Merlin AddOns 6
W Skynet SOLVED: Scrollbar Disappears on Skynet Tab on Asuswrt-Merlin 386.14 Asuswrt-Merlin AddOns 10
D Skynet Best way to configure SkyNet on RX-AT82U Asuswrt-Merlin AddOns 7
Klonoa Skynet Skynet causes router to crash and reboot Asuswrt-Merlin AddOns 1
G Skynet SkyNet and Diversion have conflicts when using user defined DNS servers - SOLVED Asuswrt-Merlin AddOns 16

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 574 (members: 25, guests: 549)
Top