Skynet Filter Validator v0.7 - Skynet Firewall Filter List IPv4 Integrity Validator

[Viktor Jaep]

Latest iteration (v0.4) is out there, including the regex fixes made by @SomeWhereOverTheRainBow that were identified by @visortgw! Thank you!

Bonus -- summary included at the end of the validation sequence:

 

[visortgw]

Latest iteration (v0.4) is out there, including the regex fixes made by @SomeWhereOverTheRainBow that were identified by @visortgw! Thank you!

Bonus -- summary included at the end of the validation sequence:

View attachment 48335

@Viktor Jaep and @SomeWhereOverTheRainBow, you both rock!

 

[SomeWhereOverTheRainBow]

@Viktor Jaep and @SomeWhereOverTheRainBow, you both rock!

I have some more optimizations I am about to give to send @Viktor Jaep way. I just fired up the ole' laptop.

 

[visortgw]

@Viktor Jaep: A suggestion for installation command string for this script (and all of your others), use "chmod 755" instead of "chmod a+rx". I am seeing the "chmod a+rx" resulting in wide open permissions (i.e., 777). Thanks.

 

[SomeWhereOverTheRainBow]

@Viktor Jaep: A suggestion for installation command string for this script (and all of your others), use "chmod 755" instead of "chmod a+rx". I am seeing the "chmod a+rx" resulting in wide open permissions (i.e., 777). Thanks.

I concur. 777 is too permissive.

 

[Ranger802004]

Maybe I'm missing something... but I'm not seeing any screenshot or other info to come to that conclusion... but I bet you're right.

Probably just need to have the script force system binaries over optional.

 

[dave14305]

Another “fix” to this bogus IP problem is to stop trusting threatview.io since most - if not all - of the “problems” have come from their feeds.

 

[John Fitzgerald]

Another “fix” to this bogus IP problem is to stop trusting threatview.io since most - if not all - of the “problems” have come from their feeds.

I get the sentiment, but how do you stop errors, that breaks functionality, of a script that's meant to help protect you.
A filter script that throws out bad entries doesn't seem like a bad idea.

Is this something that @Adamm can contribute / incorporate into fixing?

 

[dave14305]

I get the sentiment, but how do you stop errors, that breaks functionality, of a script that's meant to help protect you.
A filter script that throws out bad entries doesn't seem like a bad idea.

Is this something that @Adamm can contribute / incorporate into fixing?

I think someone should send this check script to the guy that runs threatview.io!

Certainly there can be improvements made to Skynet’s regex for IPs, but it’s also worked just fine with reputable lists.

This script itself doesn’t change how Skynet works. It will still break if a bad list is used, since it downloads the lists for itself.

 

[SomeWhereOverTheRainBow]

I get the sentiment, but how do you stop errors, that breaks functionality, of a script that's meant to help protect you.
A filter script that throws out bad entries doesn't seem like a bad idea.

Is this something that @Adamm can contribute / incorporate into fixing?

I agree. While it is highly concerning threatview.io has bad ip addresses within its list, it is equally concerning skynet has no checks in place to prevent such from breaking its usage. While we as users can easily stop using a list if a bad ip address is in the list, skynet did not make it easy to figure out/discern which list was causing the problem. Instead, skynet completed operations as if nothing was wrong. It wasn't until users read their statistics that they notice a different story- in fact statistics showed all their entries had not loaded.

 

[Viktor Jaep]

@Viktor Jaep: A suggestion for installation command string for this script (and all of your others), use "chmod 755" instead of "chmod a+rx". I am seeing the "chmod a+rx" resulting in wide open permissions (i.e., 777). Thanks.

I concur. 777 is too permissive.

Next iteration of the Skynet Filter Validator v0.5 is out there, including more optimizations and fixes by @SomeWhereOverTheRainBow... I've also made the chmod change per @visortgw's suggestion... @Adamm... please feel free to pilfer these methods if you think they can help make Skynet more solid.

 

[visortgw]

@Viktor Jaep Have you benchmarked v0.5 against v0.4? I am seeing what took about 0:21 under v0.4 now takes 1:20 for v0.5 for https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list.

 

[Viktor Jaep]

It’s from /opt/bin/grep versus /bin/grep.

Probably just need to have the script force system binaries over optional.

Trying to understand how to reduce instances of these issues with conflicting binaries... is it best practice to call "grep" in your code with an actual path, as in "/bin/grep"?

 

[Viktor Jaep]

@Viktor Jaep Have you benchmarked v0.5 against v0.4? I am seeing what took about 0:21 under v0.4 now takes 1:20 for v0.5 for https://raw.githubusercontent.com/ViktorJp/Skynet/main/filter.list.

If anything, it seems the same or faster to me. I have noticed that sometimes it will bog down on a big list, but I can't be sure if it's because it's taking longer to download it sometimes, and faster on other times... I blamed it on internet weather and how hard the list servers are being hit at that time.

 

[SomeWhereOverTheRainBow]

If anything, it seems the same or faster to me. I have noticed that sometimes it will bog down on a big list, but I can't be sure if it's because it's taking longer to curl download it than other times, where it might be downloading faster...

What might help optimize the speed is downloading the output directly to temporary file then performing operations and delete the file when finished, versus piping the output from curl which could be time consuming on larger lists.

 

[SomeWhereOverTheRainBow]

Trying to understand how to reduce instances of these issues with conflicting binaries... is it best practice to call "grep" in your code with an actual path, as in "/bin/grep"?

It won't be necessary if you are not performing operations that result in conflicting outputs.

 

[SomeWhereOverTheRainBow]

@Viktor Jaep

test using v.5

Test using v.4

here is a retest of v.4

retest of v.5 after testing v.4

another repeated run of version 5

I will do one for v.6 in a second...

Better yet, you can tell the biggest factor is the download operations.

 

[visortgw]

@Viktor Jaep

test using v.5

View attachment 48361

Test using v.4

View attachment 48362

here is a retest of v.4

View attachment 48363


retest of v.5 after testing v.4

View attachment 48365

I will do one for v.6 in a second.

Your conversion from milliseconds to seconds is off by factor of 10 (e.g., 7387 ms = 7.4 seconds) — not sure which value is correct.

 

[SomeWhereOverTheRainBow]

Your conversion from milliseconds to seconds is off by factor of 10 (e.g., 7387 ms = 7.4 seconds) — not sure which value is correct.

miliseconds is accurate, seconds is off courteous of the arithmetic feature.

 

[SomeWhereOverTheRainBow]

Your conversion from milliseconds to seconds is off by factor of 10 (e.g., 7387 ms = 7.4 seconds) — not sure which value is correct.

so I finally got it corrected. about to upload an accurate test with v.6

here is an accurate test with accurate seconds and miliseconds

second test: