Skynet Skynet - Router Firewall & Security Enhancements

[Zastoff]

Doesn't it work without that modification? Strange as my 87U does. I wonder if it has something to do with me rebooting the router after installing entware's curl as I had been doing some other script changes, maybe a reboot is needed for the entware's version to be picked up and used instead of the firmware one? It might explain Safemode's issue also.

Have tried rebooting
Get error on Consolidating Blacklist but it continues..
That do not happen when i add:

export PATH=/tmp/mnt/Zastoff/entware/bin:/sbin:/bin:/usr/sbin:/usr/bin$PATH

to script

Spoiler: Malware blacklist update without that line

Downloading filter.list | [1s]
Refreshing Whitelists | [11s]
Consolidating Blacklist | curl: option -fsLZ: is unknown
curl: try 'curl --help' for more information
[7s]
Filtering IPv4 Addresses | [6s]
Filtering IPv4 Ranges | [0s]
Applying New Blacklist | [10s]
Refreshing AiProtect Bans | [1s]
Saving Changes | [5s]

For Whitelisting Assistance -
https://www.snbforums.com/threads/r...wall-security-enhancements.16798/#post-115872
-*-

=============================================================================================================


[#] 142972 IPs (+0) -- 1638 Ranges Banned (+0) || 586 Inbound -- 0 Outbound Connections Blocked! [banmalware] [49s]

 

[dave14305]

Have tried rebooting
Get error on Consolidating Blacklist but it continues..
That do not happen when i add:

export PATH=/tmp/mnt/Zastoff/entware/bin:/sbin:/bin:/usr/sbin:/usr/bin$PATH

to script

There is a missing colon before the $PATH variable in SkyNet.

 

[dave14305]

Skynet will now require v384.13

Technically it requires 384.14 which brought 7.66.0 with the -Z option.

Working well on John’s fork with the Entware curl, but maybe a version check is needed instead of a model check since 384.14 is still new-ish. Skynet auto-update will break anyone’s malware updates if they haven’t upgraded yet.

 

[Adamm]

Technically it requires 384.14 which brought 7.66.0 with the -Z option.

Right, I misread the git commits.

Working well on John’s fork with the Entware curl, but maybe a version check is needed instead of a model check since 384.14 is still new-ish. Skynet auto-update will break anyone’s malware updates if they haven’t upgraded yet.

Technically the current checks should cover all bases (87U / 3200U / Johns fork), not sure why it seems to only be working for some users though. Doesn't make much sense

 

[SomeWhereOverTheRainBow]

Right, I misread the git commits.



Technically the current checks should cover all bases (87U / 3200U / Johns fork), not sure why it seems to only be working for some users though. Doesn't make much sense

skynet decides....

 

[Zastoff]

Right, I misread the git commits.



Technically the current checks should cover all bases (87U / 3200U / Johns fork), not sure why it seems to only be working for some users though. Doesn't make much sense

dave14305 said:
But maybe a version check is needed instead of a model check since 384.14 is still new-ish. Skynet auto-update will break anyone’s malware updates if they haven’t upgraded yet.

So if you have a router that is able to have 384.14 but still runs a older firmware version? Will those routers also auto use entware`s curl?

 

[dave14305]

How does the new curl time condition check work if you are removing the files from the lists directory before the curl command? What is it comparing with in this case?

edit: never mind, dopey me. It’s only purging if any file is missing, presumably to avoid a bad time comparison.

 

[Adamm]

So if you have a router that is able to have 384.14 but still runs a older firmware version? Will those routers also auto use entware`s curl?

Third time is the charm I guess, I added another hotfix so any model on any firmware running curl >v7.66.0 will use entware binaries if available.

How does the new curl time condition check work if you are removing the files from the lists directory before the curl command? What is it comparing with in this case?

edit: never mind, dopey me. It’s only purging if any file is missing, presumably to avoid a bad time comparison.

Usually the time-cond flag handles this automatically and downloads the file if its missing, but there was some weird behavior when mixing it with the parallel flag (I assume a bug on curl's end due to the newness of the feature) if only some files were missing they would be skipped, so I had to code a workaround that if any file was missing we wipe the entire lists directory and start fresh.

Lots of hair was pulled out trying to figure out why the curl command was only working in some cases, almost made me give up on using time-cond entirely thinking it didn't work

 

[dave14305]

Third time is the charm I guess, I added another hotfix so any model on any firmware running curl >v7.66.0 will use entware binaries if available.

Awesome. How about an ”else” in case Entware isn’t installed? Let me know when to stop being annoying.

Usually the time-cond flag handles this automatically and downloads the file if its missing, but there was some weird behavior when mixing it with the parallel flag (I assume a bug on curl's end due to the newness of the feature) if only some files were missing they would be skipped, so I had to code a workaround that if any file was missing we wipe the entire lists directory and start fresh.

Lots of hair was pulled out trying to figure out why the curl command was only working in some cases, almost made me give up on using time-cond entirely thinking it didn't work

I was reading the man page and was wondering how it was dealing with multiple -z conditions in combination with parallel.

If this option is used several times, the last one will be used.

Great work, I always learn a lot of cool things reading your code.

 

[Adamm]

Awesome. How about an ”else” in case Entware isn’t installed?

What are you suggesting goes under this else clause, a notification or something?

Let me know when to stop being annoying.

Suggestions are always welcome, quite easy to get tunnel vision and overlook issues.

I was reading the man page and was wondering how it was dealing with multiple -z conditions in combination with parallel.
Great work, I always learn a lot of cool things reading your code.

We end up with this monstrosity which is generated on-the-fly

curl -fsLZ https://iplists.firehol.org/files/alienvault_reputation.ipset -Oz alienvault_reputation.ipset https://iplists.firehol.org/files/bds_atif.ipset -Oz bds_atif.ipset https://iplists.firehol.org/files/bi_sshd_2_30d.ipset -Oz bi_sshd_2_30d.ipset https://iplists.firehol.org/files/blocklist_net_ua.ipset -Oz blocklist_net_ua.ipset https://iplists.firehol.org/files/coinbl_ips.ipset -Oz coinbl_ips.ipset https://iplists.firehol.org/files/cybercrime.ipset -Oz cybercrime.ipset https://iplists.firehol.org/files/dyndns_ponmocup.ipset -Ozdyndns_ponmocup.ipset https://iplists.firehol.org/files/et_block.netset -Oz et_block.netset https://iplists.firehol.org/files/et_compromised.ipset -Oz et_compromised.ipset https://iplists.firehol.org/files/firehol_level2.netset -Oz firehol_level2.netset https://iplists.firehol.org/files/firehol_level3.netset -Oz firehol_level3.netset https://iplists.firehol.org/files/normshield_high_attack.ipset -Oz normshield_high_attack.ipset https://iplists.firehol.org/files/normshield_high_bruteforce.ipset -Oz normshield_high_bruteforce.ipset https://iplists.firehol.org/files/ransomware_online.ipset -Oz ransomware_online.ipset https://iplists.firehol.org/files/ransomware_rw.ipset -Oz ransomware_rw.ipset https://iplists.firehol.org/files/spamhaus_edrop.netset -Oz spamhaus_edrop.netset https://iplists.firehol.org/files/urandomusto_ssh.ipset -Oz urandomusto_ssh.ipset https://iplists.firehol.org/files/urandomusto_telnet.ipset -Oz urandomusto_telnet.ipset https://iplists.firehol.org/files/urlvir.ipset -Oz urlvir.ipset https://iplists.firehol.org/files/uscert_hidden_cobra.ipset -Oz uscert_hidden_cobra.ipset

 

[dave14305]

What are you suggesting goes under this else clause, a notification or something?

That’s the tough part because you need to get their attention to install Entware if they’ve auto-updated Skynet. Maybe a notification and a syslog entry if they ever happen to look there. How would anyone know their banmalware updates were failing? All you can really assume is people have a usb with a swap file.

 

[Andy1932]

Skynet is working for me, but doesn't seem to be doing anything.

 

[dave14305]

View attachment 20394
Skynet is working for me, but doesn't seem to be doing anything.

Run 3 then update. Errors, or does it install curl and work the next time?

 

[Andy1932]

Run 3 then update. Errors, or does it install curl and work the next time?

Downloading filter.list | [0s]
Refreshing Whitelists | [3s]
Consolidating Blacklist | curl: option -fsLZ: is unknown
curl: try 'curl --help' for more information
[0s]
[*] List Content Error Detected - Stopping Banmalware

 

[dave14305]

I think @Adamm needs to avoid changing the PATH now, to avoid favoring firmware curl over Entware curl in your situation.

 

[Adamm]

That’s the tough part because you need to get their attention to install Entware if they’ve auto-updated Skynet. Maybe a notification and a syslog entry if they ever happen to look there. How would anyone know their banmalware updates were failing? All you can really assume is people have a usb with a swap file.

Good point, added.

View attachment 20394
Skynet is working for me, but doesn't seem to be doing anything.

If you have entware installed, force update Skynet to the latest commit and the issue should be resolved. Otherwise you will need to update your firmware to v384.14 to utilize the new curl version.

I think @Adamm needs to avoid changing the PATH now, to avoid favoring firmware curl over Entware curl in your situation.

I believe he is running a pre-hotfix version of Skynet. FWIW we added the path var around two years ago because I vaguely remember a specific stripped down busybox binary acting differently then the full entware version (it also makes development easier so I don't accidentally use non supported flags). Teething issues but most use cases should be covered now

 

[dave14305]

Good point, added.




If you have entware installed, force update Skynet to the latest commit and the issue should be resolved. Otherwise you will need to update your firmware to v384.14 to utilize the new curl version.



I believe he is running a pre-hotfix version of Skynet. FWIW we added the path var around two years ago because I vaguely remember a specific stripped down busybox binary acting differently then the full entware version (it also makes development easier so I don't accidentally use non supported flags). Teething issues but most use cases should be covered now

No, now I’m broken with this latest update too. Path can’t be overridden if relying on Entware curl.

[i] Downloading filter.list         | [1s]
[i] Refreshing Whitelists           | [7s]
[i] Consolidating Blacklist         | curl: option -fsLZ: is unknown
curl: try 'curl --help' for more information
[8s]
[i]

 

[Andy1932]

I thought I updated to the latest hotfix, as I forced updated after I saw your post. I'll try again.

 

[dave14305]

Or maybe only check the version of /usr/sbin/curl? Thinking out loud...

 

[Adamm]

No, now I’m broken with this latest update too. Path can’t be overridden if relying on Entware curl.

[i] Downloading filter.list         | [1s]
[i] Refreshing Whitelists           | [7s]
[i] Consolidating Blacklist         | curl: option -fsLZ: is unknown
curl: try 'curl --help' for more information
[8s]
[i]

I thought I updated to the latest hotfix, as I forced updated after I saw your post. I'll try again.

We were checking the entware binary in the last hotfix as it overrides the PATH value by default, how frustrating.

Should be fixed now guys in 7.0.2, sorry for the inconvenience!