Skynet Skynet - Router Firewall & Security Enhancements

[Adamm]

I've noticed a few problems. When I leave Skynet in debug mode for a long while, it stops using the whitelist. When I return it to vanilla afterwards, it causes the router to suddenly reboot.

I set it to update more frequently yesterday, it was in debug mode, and it corrupted the whitelist.

Sometimes I get strange messages when updating banmalware, a message about comments and ipset instead of showing the times it takes to update.

Without exact errors can't do much. Not sure exactly how the Whitelist "gets corrupt" either, debug mode is simply just an 1 extra IPTable rule that announces the blocks in syslog while dropping them. Need more info

 

[john9527]

Is there anything we can do or I can do?

In the short term, add a sleep so they stay out of each others way like you have done.

In the long term, possibly add some similar error checking/retries to both Skynet and updown.sh to handle the collision.

 

[G F]

If anyone's interested, here's the custom list I use for Skynet. https://pastebin.com/raw/ZhT8ckH9 I would highly suggest whitelisting the IPs you use in your router(s) before using it.

Interested to know a bit more information about this custom list. Do you keep manually importing them via ' sh /jffs/scripts/firewall import' or are you using another method to keep them updated on a schedule?

 

[Adamm]

Interested to know a bit more information about this custom list. Do you keep manually importing them via ' sh /jffs/scripts/firewall import' or are you using another method to keep them updated on a schedule?

Its a banmalware custom filter list to replace the default one. Very similar although alittle more aggressive with a few added lists.

 

[MacG32]

Interested to know a bit more information about this custom list. Do you keep manually importing them via ' sh /jffs/scripts/firewall import' or are you using another method to keep them updated on a schedule?

I use sh /jffs/scripts/firewall banmalware https://pastebin.com/raw/ZhT8ckH9 and have it set to the default update. It will now keep updating it nightly, just like the default list.

 

[2992]

@Adamm, just reporting, something has happen, but I do not know what:

Dec 1 02:00:07 Skynet: [Complete] 132873 IPs / 1944 Ranges Banned. 0 New IPs / 0 New Ranges Banned. Inbound / Outbound Connections Blocked! [7s]
Dec 1 02:25:47 Skynet: [ERROR] 404 Error Detected - Stopping Banmalware
Dec 1 03:00:07 Skynet: [Complete] 132873 IPs / 1944 Ranges Banned. 0 New IPs / 0 New Ranges Banned. Inbound / Outbound Connections Blocked! [7s]

 

[Adamm]

@Adamm, just reporting, something has happen, but I do not know what:

As the error suggests, the link you provided for custom filter list is 404'ing

 

[2992]

As the error suggests, the link you provided for custom filter list is 404'ing

I have no custom filter list, just default configuration.

 

[Adamm]

I have no custom filter list, just default configuration.

Same logic applies, for whatever reason the github maybe have been temporarily unresponsive. If you try run the command again it should continue as normal.

 

[2992]

Same logic applies, for whatever reason the github maybe have been temporarily unresponsive. If you try run the command again it should continue as normal.

ok, got it, it's clear now, thanks much!

 

[MacG32]

There seems to be a RAM issue when running Skynet with this router. Preinstall RAM usage is 51-52%. Installed and running with my filter list, 68%. Temporary disabled, 57%. Restarted, 81%. Reboot router, 91%. Run Skynet in SSH after the reboot, RAM usage drops to 68% again. If I reboot, then try to update banmalware, it automatically reboots. When I switch between debug after some time and back to vanilla, it automatically reboots. Sometimes when it updates banmalware, it reboots. I don't know if @RMerlin could shed some light on this. I've gotten 2 big errors (one reported) where ipset was involved. I've tested all of this on the release firmware and the alphas. It just doesn't function properly on this router. It's even corrupted the router on reboot to where it doesn't allow internet access through it. I've had to reflash and restore to defaults many times after automatic reboots. I could access the router and could see it had an IP address from my ISP, but watched the NTP process try to sync the time over and over again and not have internet access to any attached devices without anything showing in the log.

 

[Adamm]

There seems to be a RAM issue when running Skynet with this router. Preinstall RAM usage is 51-52%. Installed and running with my filter list, 68%. Temporary disabled, 57%. Restarted, 81%. Reboot router, 91%. Run Skynet in SSH after the reboot, RAM usage drops to 68% again. If I reboot, then try to update banmalware, it automatically reboots. When I switch between debug after some time and back to vanilla, it automatically reboots. Sometimes when it updates banmalware, it reboots. I don't know if @RMerlin could shed some light on this. I've gotten 2 big errors (one reported) where ipset was involved. I've tested all of this on the release firmware and the alphas. It just doesn't function properly on this router. It's even corrupted the router on reboot to where it doesn't allow internet access through it. I've had to reflash and restore to defaults many times after automatic reboots. I could access the router and could see it had an IP address from my ISP, but watched the NTP process try to sync the time over and over again and not have internet access to any attached devices without anything showing in the log.

I think the ram issue is more with the AC86U in general then being caused by Skynet directly. Skynet during its most intensive processes only temporarily uses less then 20-30mb of ram. With a large IPSet loaded it idles around 10mb usage (atleast on my AC68U). I even implement things like clearing the cache every time Skynet is run to try prevent this.

I know @thelonelycoder also had some issues with DNSMasq running out of ram causing issues on this model. I'm not sure there is much that can be done from my end as this is from what I can see an issue with the model in general having very little spare resources. I hope this is something Asus address in future updates along with the fork() issues. I'm in the process of trying to get a AC86U, so hopefully I will be able to investigate these issues first hand in the near future.

Edit; I also remember you use a custom banmalware filter, do note that more IPSet entries = more ram usage, so this could be potentially using more of the (limited) available resources. The default list currently has around 140k entries and uses 7.6mb of ram (plus overheads of running the kernel module I assume)

 

[thelonelycoder]

I think the ram issue is more with the AC86U in general then being caused by Skynet directly. Skynet during its most intensive processes only temporarily uses less then 20-30mb of ram. With a large IPSet loaded it idles around 10mb usage (atleast on my AC68U). I even implement things like clearing the cache every time Skynet is run to try prevent this.

I know @thelonelycoder also had some issues with DNSMasq running out of ram causing issues on this model. I'm not sure there is much that can be done from my end as this is from what I can see an issue with the model in general having very little spare resources. I hope this is something Asus address in future updates along with the fork() issues. I'm in the process of trying to get a AC86U, so hopefully I will be able to investigate these issues first hand in the near future.

Edit; I also remember you use a custom banmalware filter, do note that more IPSet entries = more ram usage, so this could be potentially using more of the (limited) available resources. The default list currently has around 140k entries and uses 7.6mb of ram (plus overheads of running the kernel module I assume)

I'm going to use your swap file creation tool to be included in AB-Solution and amtm. Both will feature the same management options.

AB-Solution will urge you to use a swap file with the next update. With the additional hosts files @Adamm and I discussed to use in AB instead of Skynet, it will bring most routers to its limit during the update when a large blocking file is selected. Especially so if IPv6 is enabled.

 

[elorimer]

After searching logs and experimenting with openvpn server I have the opinion that skynet starts a little to soon in my routers boot process. It interferes with the "up-down command" during openvpn server's launch then exits with fatal error. If I use a restart script in services-start openvpn server starts no problem. This brings me to my question. Is there a way to delay skynet from starting by say 10 or 15 seconds?

I have this problem too. 87R, with ab-solution, (pixelserv-tls 2.0RC1) and skynet all up to date. My openvpn servers fail at the updown.sh script.

Also, I've seen ab-solution hang at "checking installation state", and the pixelserv stat's page be unresponsive. Other pages take minutes to load. With one exception this all occurred after being away from the router for 10 days or so, and updated it by adding amtm, updating pixelserve from 8d to RC1, skynet from 5.5.4 to 5.5.5 and ab-solution to 3.10. The exception being that after being up for 8 days, the openvpn server kicked off.

I noticed a new script, openvpn-event, with:

#!/bin/sh
sh /jffs/scripts/firewall whitelist vpn # Skynet Firewall Addition

Not sure when that was introduced.

I've added sleep 30 in the firewall-start script, which allows the openvpn servers to start on a reboot. openvpn completes about 16 seconds before skynet starts.

I have some debugging to do to narrow this down.

 

[Adamm]

My openvpn servers fail at the updown.sh script.

I did investigate this briefly when it was first reported a few days ago, I looked at updown.sh and couldn't see any ways it would conflict with Skynet. They dont use the same IPTables tables or chains, and beyond that everything else Skynet does is independent. My assumption was that its updown.sh failing possibly for another reason (maybe it doesn't handle high CPU usage well?). If you can give me some sort of configuration I can reproduce on my end as I've never used updown.sh I can look into it further. If its as simple as needing a delay, I can add that to the startup function if updown usage is detected.

Not sure when that was introduced.

Few weeks ago I added this with the extra VPN support to make sure any server updates were caught and whitelisted. Its a fairly basically command and just whitelists VPN related nvram values.

 

[elorimer]

If you can give me some sort of configuration I can reproduce on my end as I've never used updown.sh I can look into it further.

Just enable the two openvpn servers, I think.

 

[elorimer]

My assumption was that its updown.sh

So, updown.sh in both etc/openvpn/server1 and server2 are links to this openvpn-event script with only the skynet addition. If I disable the servers, those directories disappear; if I enable them again I get the links back. Are you overwriting something when you add that whitelisting line?

EDIT: I uninstalled skynet and rebooted. Now updown.sh links to openvpn-event, but that is empty except for a shebang.

 

[Adamm]

Are you overwriting something when you add that whitelisting line?

No, it doesn't touch them at all. That entry is only created during the install function, and it simply appends a line to the end of the file. Beyond that its never touched or verified if it exists still.

 

[elorimer]

I'm so confused. I disabled both servers and rebooted with skynet uninstalled. Then I enabled both servers. Updown.sh doesn't get created then, and neither does openvpn-event. Is updown.sh a feature of skynet?

 

[Adamm]

I'm so confused. I disabled both servers and rebooted with skynet uninstalled. Then I enabled both servers. Updown.sh doesn't get created then, and neither does openvpn-event. Is updown.sh a feature of skynet?

No, Skynet doesn't use this file at all. Skynets VPN interaction is limited to adding 1 line to openvpn-event which preforms a few basic "nvram get xxx" commands to whitelist.