Skynet Skynet - Router Firewall & Security Enhancements

[Adamm]

Maybe we can change "curl -fs" to "curl -sS" so we can see it's throwing an error?

If you want to go ahead and change this manually on line 1274 and see if we get any extra output that would be great.

 

[SeaConn]

go ahead and change this manually on line 1274

Okay. Did just that. Changed it to "curl -sS". About one out of three times I get this if I'm quick.

Saving Changes                  [1s]
Removing Previous Malware Bans  [1s]
Downloading filter.list         [0s]
Whitelisting Shared Domains     firewall: line 2046: date: Cannot allocate memory
[-1509523742s]
Consolidating Blacklist         [4s]
Filtering IPv4 Addresses        [1s]
Filtering IPv4 Ranges           [0s]
Applying Blacklists             [2s]

For False Positive Website Bans Use; ( sh firewall whitelist domain URL )

Skynet: [Complete] 97883 IPs / 3097 Ranges Banned. -53612 New IPs / 0 New Ranges Banned.  Inbound /  Outbound Connections Blocked! [22s]

The rest of the time it finishes fine. In fact, I noticed I only get the memory error if I run it quickly twice in a row. If I wait five minutes between updating it runs fine. Garbage collector maybe? No clue.

Either way, I don't think 382.1 likes the "-fs" option with curl.

 

[Adamm]

Okay. Did just that. Changed it to "curl -sS". About one out of three times I get this if I'm quick.

Saving Changes                  [1s]
Removing Previous Malware Bans  [1s]
Downloading filter.list         [0s]
Whitelisting Shared Domains     firewall: line 2046: date: Cannot allocate memory
[-1509523742s]
Consolidating Blacklist         [4s]
Filtering IPv4 Addresses        [1s]
Filtering IPv4 Ranges           [0s]
Applying Blacklists             [2s]

For False Positive Website Bans Use; ( sh firewall whitelist domain URL )

Skynet: [Complete] 97883 IPs / 3097 Ranges Banned. -53612 New IPs / 0 New Ranges Banned.  Inbound /  Outbound Connections Blocked! [22s]

The rest of the time it finishes fine. In fact, I noticed I only get the memory error if I run it quickly twice in a row. If I wait five minutes between updating it runs fine. Garbage collector maybe? No clue.

Either way, I don't think 382.1 likes the "-fs" option with curl.

If you look at the ram graph on the router GUI do you notice any excessive spikes when running the command? On my AC68U the most I see is a brief 20MB increase

 

[SeaConn]

When I just ran it, the ram went from 383 to 404 (spike) back down to 393.

Ran Banmalware again and got the fork error, and the ram dropped to 381 MB.

Ran it again and this time it finished without error and is using 391 MB.

Ran it again and it only loaded 29,000 IPs, using 383 MB.

Ran it again and got the fork error. Ram dropped to 381 MB.

Ran it one last time and it is now using 393 MB and banning 118,000 IPs.

I think it's flooding the ram above its physical limit. The ram spike is causing the issue. A big chunk of the 512 MB it has is reserved for System uses, so I guess it makes sense that somewhere around 4xx it shirts the bed. When it has 29,000 IPs already loaded up (not sure why that's happening) and it downloads 118,000 more for comparison/addendum, it gets bumped over the threshold and throws the memory or fork error, depending on where it was in the process.

All speculation, of course. I'm as likely to be wrong as I am right.

 

[Adamm]

When I just ran it, the ram went from 383 to 404 (spike) back down to 393.

Ran Banmalware again and got the fork error, and the ram dropped to 381 MB.

Ran it again and this time it finished without error and is using 391 MB.

Ran it again and it only loaded 29,000 IPs, using 383 MB.

Ran it again and got the fork error. Ram dropped to 381 MB.

Ran it one last time and it is now using 393 MB and banning 118,000 IPs.

I think it's flooding the ram above its physical limit. The ram spike is causing the issue. A big chunk of the 512 MB it has is reserved for System uses, so I guess it makes sense that somewhere around 4xx it shirts the bed. When it has 29,000 IPs already loaded up (not sure why that's happening) and it downloads 118,000 more for comparison/addendum, it gets bumped over the threshold and throws the memory or fork error, depending on where it was in the process.

All speculation, of course. I'm as likely to be wrong as I am right.

I'm surprised the AC86U has such a high idle usage. My AC68U idles around 80MB with all the bells and whistles. I also saw @.TT. having similar ram issues with AB-Solution.

When running top (then press m to sort by memory usage) what does the output look like, maybe there is something else with a memory leak?

 

[thelonelycoder]

I'm surprised the AC86U has such a high idle usage. My AC68U idles around 80MB with all the bells and whistles. I also saw @.TT. having similar ram issues with AB-Solution.

When using a larger blocking file in AB-Solution then more RAM is used by the file.
I have about 45% idle usage. When running a Skynet update for example, I get spikes up to 60%, so about 15% are used by Skynet a short while.
This can easily bring ram usage to the limit with a large blocking file.
I use swap files on all routers, just to be on the safe side and make things a tad snappier.
For Howto swapfile look here: https://www.snbforums.com/threads/b...1-beta-is-available.41661/page-15#post-355902

 

[.TT.]

@Adamm
Added swap as @thelonelycoder recommended and banmalware is now indeed updating.
Thank you both for all your help!

 

[Adamm]

While a swap-file may be a short term solution here, I still think there's got to be a better way. The fact Asus's newest model with twice as much ram runs out of memory with a simple curl process is hard to fathom. Weather that be trying to reduce the staggering idle usage which is x5 more then previous models or correcting a possible memory leak I don't know. But I can't imagine Asus's engineers would leave themselves with such little resources for future updates/features or even under high load situations.

 

[Adamm]

I've pushed another temporary fix for AC86U users (it will require a forced update). This time freeing pagecache, dentries and inodes prior to issuing the curl command that keeps seeming to fail.

Not ideal, but for the time being it may free enough ram to keep things moving until the overall issue is resolved.

 

[SeaConn]

When using a larger blocking file in AB-Solution then more RAM is used by the file.

I was wondering how AB-Solution never crashed my RT-AC66U. Beautiful.

 

[SeaConn]

Not ideal, but for the time being it may free enough ram to keep things moving until the overall issue is resolved.

I still get the curl error when running option 3. Oh well. Thanks for trying -- it's still in Beta and will hopefully get cleaned up down the road.

Any idea why 382.1 is such a resource hog? I could live without half the functions Asus has crammed into the GPL. WTFast? My initial thought was WTF am I going to use that for? Be nice if we could opt out.

For posterity, my memory usage looks like this with everything loaded:

User0@RT-AC86U-98E0:/tmp/home/root# free -m
             total       used       free     shared    buffers     cached
Mem:        440372     329248     111124          0       1488      46572
-/+ buffers/cache:     281188     159184
Swap:            0          0          0

 

[.TT.]

I still get the curl error when running option 3. Oh well. Thanks for trying -- it's still in Beta and will hopefully get cleaned up down the road.

Any idea why 382.1 is such a resource hog? I could live without half the functions Asus has crammed into the GPL. WTFast? My initial thought was WTF am I going to use that for? Be nice if we could opt out.

For posterity, my memory usage looks like this with everything loaded:

User0@RT-AC86U-98E0:/tmp/home/root# free -m
             total       used       free     shared    buffers     cached
Mem:        440372     329248     111124          0       1488      46572
-/+ buffers/cache:     281188     159184
Swap:            0          0          0

Looks like your swap is incorrect. 0b total.

 

[RMerlin]

Any idea why 382.1 is such a resource hog?

On the Broadcom HND platform, Broadcom reserves 13% of the total RAM for buffers that are used by its various components, for things like buffers and such. So out of 512 MB, only 440 MB remains available to the rest of the system.

The kernel is 64-bit, so it means its modules will also have a slightly higher memory usage than older 32-bit platforms.

Note that a service like WTFast doesn't require any memory at all unless you enable it. It only uses some flash space, and there's still a good amount of flash available for future usage.

 

[Andy1932]

I've pushed another temporary fix for AC86U users (it will require a forced update). This time freeing pagecache, dentries and inodes prior to issuing the curl command that keeps seeming to fail.

Not ideal, but for the time being it may free enough ram to keep things moving until the overall issue is resolved.

How do I install this fix? I tried:

/usr/sbin/wget -O /jffs/scripts/firewall https://github.com/Adamm00/IPSet_ASUS/blob/daac792db0454d8d32e7497be40b3079c8e857bb/firewall.sh
chmod +x /jffs/scripts/firewall
sh /jffs/scripts/firewall install

 

[SeaConn]

Looks like your swap is incorrect. 0b total.

I've attempted setting up a swap previously on the ac66u and it didn't end well. My current USB drive is formatted to have 512 MB Linux swap at the beggining and then the rest is formatted as ext3, about 1.5 GB. Can you point me to a guide?

 

[skeal]

How do I install this fix? I tried:

/usr/sbin/wget -O /jffs/scripts/firewall https://github.com/Adamm00/IPSet_ASUS/blob/daac792db0454d8d32e7497be40b3079c8e857bb/firewall.sh
chmod +x /jffs/scripts/firewall
sh /jffs/scripts/firewall install

You should be able to force an update by using:

sh /jffs/scripts/firewall update -f

 

[Andy1932]

You should be able to force an update by using:

sh /jffs/scripts/firewall update -f

Thanks for the tip. That worked, but I'm still getting this:

Router Model; RT-AC86U
Skynet Version; v5.4.4 (2/11/2017)
iptables v1.4.14 - (eth0 @ 192.168.1.1)
ipset v6.32, protocol version: 6
FW Version; 382.1_beta2-gc21d7dd (Oct 24 2017) (4.1.27)
Install Dir; /tmp/mnt/disk/skynet (14.3G Space Available)
Boot Args; /jffs/scripts/firewall start banmalware autoupdate usb=/tmp/mnt/disk

Checking Skynet IPTable...                              [Failed]

 

[Butterfly Bones]

I've attempted setting up a swap previously on the ac66u and it didn't end well. My current USB drive is formatted to have 512 MB Linux swap at the beggining and then the rest is formatted as ext3, about 1.5 GB. Can you point me to a guide?

I used this as a part of installing Entware. The setup includes a swap file.
https://www.hqt.ro/how-to-install-new-generation-entware/

I'm getting a 504 gateway timeout on the link right now, it sometimes goes down. (shrug)
Here are the commands - Entware-ng install

cd /tmp
wget -c -O entware-ngu-setup.sh http://goo.gl/hshQkA

chmod +x ./entware-ngu-setup.sh

./entware-ngu-setup.sh

I have Entware, AB-Solution, Skynet, DNSCrypt on a single partition USB stick, plus a few entware packages that were installed with Entware, plus the SFTP server.

 

[SeaConn]

Thanks for the tip. That worked, but I'm still getting this:

Router Model; RT-AC86U
Skynet Version; v5.4.4 (2/11/2017)
iptables v1.4.14 - (eth0 @ 192.168.1.1)
ipset v6.32, protocol version: 6
FW Version; 382.1_beta2-gc21d7dd (Oct 24 2017) (4.1.27)
Install Dir; /tmp/mnt/disk/skynet (14.3G Space Available)
Boot Args; /jffs/scripts/firewall start banmalware autoupdate usb=/tmp/mnt/disk

Checking Skynet IPTable...                              [Failed]

I get this:

User0@RT-AC86U-98E0:/tmp# sh /jffs/scripts/firewall install
/jffs/scripts/firewall: line 7: syntax error: unexpected newline

 

[RMerlin]

I just ran 35 parallel curl sessions here, without any problem.

You might want to limit the number of concurrent sessions however - firing 32 parallel connections at that one single server might eventually get you blacklisted for abuse. 4 is considered more polite, 8 would be the absolute limit.