Skynet Filter Validator v0.7 - Skynet Firewall Filter List IPv4 Integrity Validator

[Ubimo]

Just a thought:
Is it somehow possible for you guys, @Viktor Jaep and @SomeWhereOverTheRainBow , to process/validate the lists before publishing them on github?
Shift the process somehow to an online "server"? Once a day?
https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

 

[SomeWhereOverTheRainBow]

Just a thought:
Is it somehow possible for you guys, @Viktor Jaep and @SomeWhereOverTheRainBow , to process/validate the lists before publishing them on github?
Shift the process somehow to an online "server"? Once a day?
https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list
https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

No need to now. Skynet already incorporates regex to stop use of bad and private ip addresses.

 

[John Fitzgerald]

No need to now. Skynet already incorporates regex to stop use of bad and private ip addresses.

@Ubimo has Starlink and it uses a Private IP, will Skynet not work anymore with Starlink (break)? see comment by Adamm below reference.

See discusssion here, post #374 forward: https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/page-19

See post #386 by @Adamm

Thanks.

 

[SomeWhereOverTheRainBow]

@Ubimo has Starlink and it uses a Private IP, will Skynet not work anymore with Starlink (break)? see comment by Adamm below reference.

See discusssion here, post #374 forward: https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/page-19

See post #386 by @Adamm

Thanks.

Skynet filters private out of the actual list used in the ipset. Don't just take my word for it, it is in the actual skynet code.

https://github.com/Adamm00/IPSet_ASUS/blob/master/firewall.sh#L3476

Notice the | Filter_PrivateIP pipe?

@Adamm didn't just place that there because he thinks it looks cool.

 

[John Fitzgerald]

Skynet filters private out of the actual list used in the ipset. Don't just take my word for it, it is in the actual skynet code.

https://github.com/Adamm00/IPSet_ASUS/blob/master/firewall.sh#L3476

Notice the | Filter_PrivateIP pipe?

@Adamm didn't just place that there because he thinks it looks cool.

I would like to give you an extra like, but the system only allows only one.

So, Thank You!

 

[Mogsy]

@Ubimo has Starlink and it uses a Private IP, will Skynet not work anymore with Starlink (break)? see comment by Adamm below reference.

See discusssion here, post #374 forward: https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/page-19

See post #386 by @Adamm

Thanks.

I'm behind CG-Nat too. No problem using Skynet

 

[John Fitzgerald]

I'm behind CG-Nat too. No problem using Skynet

But does yours show in RED? (the IP Address) like POST #374 & POST #382, Here: https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/page-19

Also, who are using for a provider, as I'm looking into T-Moble Home Internet Service for a family member, and trying to anticipate problems and my own learning curve?

Any insight would be great.

 

[Mogsy]

But does yours show in RED? (the IP Address) like POST #374 & POST #382, Here: https://www.snbforums.com/threads/is-default-firewall-good-enough.76648/page-19

Also, who are using for a provider, as I'm looking into T-Moble Home Internet Service for a family member, and trying to anticipate problems and my own learning curve?

Any insight would be great.

Yes it is. Skynet even give me a warning but it works.

Mar 4 13:46:23 Skynet: [*] Private WAN IP Detected 100.66.x.xx - Please Put Your Modem In Bridge Mode / Disable CG-NAT

 

[Viktor Jaep]

One last hurrah for Filter Validator with a final release to v0.7... just wanted to add the necessary verbiage to indicate here in this thread and in the script that Skynet v7.3.6 contains all the necessary regex fixes (with many thanks to @SomeWhereOverTheRainBow for that!), which were wholly inspired by Filter Validator. This tool will still be able to play some valuable roles:

• Determine if your blacklist operators are actually maintaining their lists with valid IPs... ones that don't (perhaps, like Threatview), should be considered "fringe", and may cause more issues down the road for you.
• Indicate how many entries are in each of the blacklist(s)... know that Skynet does have a max limit of 500,000... and this will give you some indication if you're getting close to this max. (thought Skynet may also do some form of dedupe as well to bring that number down a bit)
• Determine if blacklist operators are still supporting their lists... some may drop off the planet, or no longer exist. You would get no indication of this happening within Skynet, but Filter Validator will definitely let you know as shown below:

 

[dave14305]

They could probably use some help on OpenWrt.

banIP support thread

Actually the list includes an invalid IP address (line 3, columns 6712-6724) which can't be parsed by nft ... looking in the temp. load file reveals the culprit: 198.265.75.69 Please report it upstream to the list/feed maintainer.

forum.openwrt.org

 

[Viktor Jaep]

They could probably use some help on OpenWrt.

banIP support thread

Actually the list includes an invalid IP address (line 3, columns 6712-6724) which can't be parsed by nft ... looking in the temp. load file reveals the culprit: 198.265.75.69 Please report it upstream to the list/feed maintainer.

forum.openwrt.org

Woof... That sounds like opening a big can of worms! Lol

 

[Phil Mcavity]

One last hurrah for Filter Validator with a final release to v0.7... just wanted to add the necessary verbiage to indicate here in this thread and in the script that Skynet v7.3.6 contains all the necessary regex fixes (with many thanks to @SomeWhereOverTheRainBow for that!), which were wholly inspired by Filter Validator. This tool will still be able to play some valuable roles:

• Determine if your blacklist operators are actually maintaining their lists with valid IPs... ones that don't (perhaps, like Threatview), should be considered "fringe", and may cause more issues down the road for you.
• Indicate how many entries are in each of the blacklist(s)... know that Skynet does have a max limit of 500,000... and this will give you some indication if you're getting close to this max. (thought Skynet may also do some form of dedupe as well to bring that number down a bit)
• Determine if blacklist operators are still supporting their lists... some may drop off the planet, or no longer exist. You would get no indication of this happening within Skynet, but Filter Validator will definitely let you know as shown below:

View attachment 48428

That really is awesome. Thank you guys for your work

 

[Ubimo]

I noticed a big decline in IPs from this list: (now ~216.000 IPs, some days ago ~400.000 IPs)

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

Is this list still "good"? Or do you recommend another list/link?
Thanks in advance!

 

[Viktor Jaep]

I noticed a big decline in IPs from this list: (now ~216.000 IPs, some days ago ~400.000 IPs)

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

Is this list still "good"? Or do you recommend another list/link?
Thanks in advance!

It looks like this one blacklist (https://myip.ms/files/blacklist/general/latest_blacklist.txt) contains IP6 entries, which may be gumming them up... not sure if Skynet filters for those, though I could have sworn it did... perhaps @SomeWhereOverTheRainBow can give us his take?

 

[SomeWhereOverTheRainBow]

It looks like this one blacklist (https://myip.ms/files/blacklist/general/latest_blacklist.txt) contains IP6 entries, which may be gumming them up... not sure if Skynet filters for those, though I could have sworn it did... perhaps @SomeWhereOverTheRainBow can give us his take?

Skynet filter does not actually use any ipv6 entries. So the only thing skynet would catch is ipv4 entries. It is quite possible, the creators of that list aggregated some of the ipv4 entries, and some of the single ip addresses are now grouped as ranges. If this were the case, that would explain why such a large drop in addresses.

 

[SomeWhereOverTheRainBow]

I noticed a big decline in IPs from this list: (now ~216.000 IPs, some days ago ~400.000 IPs)

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/filter.list

Is this list still "good"? Or do you recommend another list/link?
Thanks in advance!

You should give this list a try

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

 

[Ubimo]

You should give this list a try

https://raw.githubusercontent.com/jumpsmm7/GeneratedAdblock/master/myfilter.list

Thanks, but unfortunately this list blocked my e-mail and music stream.

 

[SomeWhereOverTheRainBow]

Thanks, but unfortunately this list blocked my e-mail and music stream.

That's strange. I wonder if they are commonly mistaken, or have ever been used as an attack vector then. Sometimes honest services end up on lists because they are easily exploited , or have been associated with reputation issues in the past. Hard to say.

 

[Ubimo]

The list I mentioned above now has again ~400000 entries.

 

[SomeWhereOverTheRainBow]

The list I mentioned above now has again ~400000 entries.

Also keep track of the number of blocked ranges changes. A 10k increase in blocked ranges can easily explain a 100k drop in single IP entries, and vice versa.